Scraping
Scraping
Every field on POST /scrape, and which engine reads it.
POST /scrape takes a JSON body. Only url and proxy_type are required.
Two engines, one request
render_js picks the engine, and the engine decides which other fields mean
anything.
render_js: true(the default) runs a real browser. Use it when the page builds itself client-side, or when you need a screenshot or interaction.render_js: falsesends a plain HTTP request wearing a real browser’s TLS and HTTP fingerprint. Much cheaper and much faster. Use it whenever the HTML you want is in the first response.
Fields on both engines
| Field | Type | Notes |
|---|---|---|
url | string | Required. Validated against SSRF: private and non-HTTP addresses are refused. |
proxy_type | datacenter | premium | Required. premium is residential. |
render_js | boolean | Default true. |
country_code | string | ISO 3166-1 alpha-2. Comma-separate up to 10 (us,gb). See Proxies. |
proxy_session | string | Sticky session id, letters and digits, up to 32. |
proxy_session_lifetime | string | 30s, 30m, 2h. Requires proxy_session. |
headers | object | Added to the engine’s own. Yours win on a collision. |
cookies | array | Cookie objects sent with the request. |
session_id | string | Reuse a stored browser session. 1–128 of [A-Za-z0-9._-]. |
account_id | string | Use a managed login instead. Mutually exclusive with session_id. |
Browser only (render_js: true)
| Field | Type | Notes |
|---|---|---|
wait_until | string | Defaults to domcontentloaded if you omit it. |
wait | integer | Extra milliseconds to wait, 0–30000. |
wait_for | string | A selector to wait for. |
viewport_width | integer | |
viewport_height | integer | |
block_resources | boolean | Skip images and fonts. Faster and cheaper on bandwidth. |
screenshot | full | visible | |
js_actions | array | Click, fill, scroll, wait, evaluate. |
HTTP only (render_js: false)
| Field | Type | Notes |
|---|---|---|
method | GET POST PUT PATCH DELETE HEAD | Default GET. |
body / json / form | string / any / object | Mutually exclusive. Sending two returns bad_request. |
header_order | array | Pins on-the-wire header order — a fingerprinting detail. |
follow_redirects | boolean | Default true. |
timeout_ms | integer | Caps the whole request. |
The response
{
"ok": true,
"url": "https://example.com/orders",
"final_url": "https://example.com/orders",
"status": 200,
"headers": { "content-type": "text/html; charset=utf-8" },
"cookies": [],
"body": "<!doctype html>...",
"body_length": 48213,
"body_encoding": "utf-8",
"truncated": false,
"content_type": "text/html; charset=utf-8",
"elapsed_ms": 1840,
"blocked": false,
"block_reason": null,
"redirects": []
}
Two fields are easy to miss:
body_encodingisutf-8orbase64. Binary responses, and text we could not decode, come back base64. Check it before parsing.truncatedmeans the body hit the size cap and was cut.body_lengthis the byte length after truncation.
When ok is false, the shape is different — see Errors.