[SCR-583] Use header-based authorization + document all API params - #23
Open
sahilsunny wants to merge 3 commits into
Open
[SCR-583] Use header-based authorization + document all API params#23sahilsunny wants to merge 3 commits into
sahilsunny wants to merge 3 commits into
Conversation
Auth: the client now sends the API key as an `Authorization: Bearer` request header instead of the `api_key` query parameter, keeping keys out of URLs and server logs. The server has accepted header auth on all endpoints since SCR-463 (June 2026) and gives the header precedence, so this is backward-compatible with all supported API versions. Docs: add complete request-parameter tables for every endpoint (HTML, Google Search, Fast Search, Amazon x3, Walmart x2, YouTube x3, ChatGPT, Gemini), sourced from the server-side param schemas. This includes `tag` (accepted everywhere), `custom_google` (still required for Google URLs — the API changelog's "removed" note is incorrect), `nb_results`, `autoselect_variant`, and deprecated-but-accepted params, marked as such. Also documents Auto-Mode's transparent_status_code exclusion and the Usage API rate limit. Tests: all client assertions moved from api_key-in-params to the Authorization header; suite fails against the old client and passes against the new one. Verified live against production: usage, HTML API, Auto-Mode, fast_search all 200 with the key absent from the URL, and a wrong key returns 401. Version: 2.1.1 -> 2.2.0.
- Remove proxy_mode from the README: it is in the server's private schema, and documenting it publicly invites an API-key leak (with proxy_mode + forward_headers the server forwards the raw Authorization header to the scraped site). - Fix swapped types (Google min_price/max_price are floats, radius is an int), correct the Amazon country note (400 when it matches the domain country, not a silent no-op), annotate missing defaults, exclude Usage from the tag note, and add renewal_subscription_date to the usage sample. - Send the ScrapingBee-Python User-Agent on all endpoints (previously only HTML API requests had it). - Add auth invariant tests: every method must authenticate via the Bearer header with no api_key query param, and a caller-supplied Authorization header cannot clobber authentication (it forwards as Spb-Authorization).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Moves auth from the
api_keyquery param to theAuthorization: Bearerheader — keeps keys out of URLs and server logs. The API has supported this on all endpoints since SCR-463, and the header wins when both are sent, so nothing breaks. No interface change:ScrapingBeeClient(api_key=...)works exactly as before.While at it, the README now documents every request parameter for every endpoint, taken straight from the server-side schemas. That includes
tag(accepted everywhere),custom_google(still required for Google URLs — the API changelog note about its removal is inaccurate),nb_results,autoselect_variant, and the deprecated-but-still-accepted params, marked as such.Tested: full suite updated (fails against the old client, passes against the new), plus live runs against production — usage, HTML API, Auto-Mode and fast_search all return 200 with the key present only in the header, and a wrong key gets a 401.
Version: 2.1.1 → 2.2.0