Scraping

Scraping

Every field on POST /scrape, and which engine reads it.

POST /scrape takes a JSON body. Only url and proxy_type are required.

Two engines, one request

render_js picks the engine, and the engine decides which other fields mean anything.

  • render_js: true (the default) runs a real browser. Use it when the page builds itself client-side, or when you need a screenshot or interaction.
  • render_js: false sends a plain HTTP request wearing a real browser’s TLS and HTTP fingerprint. Much cheaper and much faster. Use it whenever the HTML you want is in the first response.

Fields belonging to the other engine are silently dropped, not rejected. Sending screenshot with render_js: false returns a normal response with no screenshot in it, and no error explaining why. Check render_js first when a field appears to do nothing.

Fields on both engines

FieldTypeNotes
urlstringRequired. Validated against SSRF: private and non-HTTP addresses are refused.
proxy_typedatacenter | premiumRequired. premium is residential.
render_jsbooleanDefault true.
country_codestringISO 3166-1 alpha-2. Comma-separate up to 10 (us,gb). See Proxies.
proxy_sessionstringSticky session id, letters and digits, up to 32.
proxy_session_lifetimestring30s, 30m, 2h. Requires proxy_session.
headersobjectAdded to the engine’s own. Yours win on a collision.
cookiesarrayCookie objects sent with the request.
session_idstringReuse a stored browser session. 1–128 of [A-Za-z0-9._-].
account_idstringUse a managed login instead. Mutually exclusive with session_id.

Browser only (render_js: true)

FieldTypeNotes
wait_untilstringDefaults to domcontentloaded if you omit it.
waitintegerExtra milliseconds to wait, 0–30000.
wait_forstringA selector to wait for.
viewport_widthinteger
viewport_heightinteger
block_resourcesbooleanSkip images and fonts. Faster and cheaper on bandwidth.
screenshotfull | visible
js_actionsarrayClick, fill, scroll, wait, evaluate.

HTTP only (render_js: false)

FieldTypeNotes
methodGET POST PUT PATCH DELETE HEADDefault GET.
body / json / formstring / any / objectMutually exclusive. Sending two returns bad_request.
header_orderarrayPins on-the-wire header order — a fingerprinting detail.
follow_redirectsbooleanDefault true.
timeout_msintegerCaps the whole request.

The response

{
  "ok": true,
  "url": "https://example.com/orders",
  "final_url": "https://example.com/orders",
  "status": 200,
  "headers": { "content-type": "text/html; charset=utf-8" },
  "cookies": [],
  "body": "<!doctype html>...",
  "body_length": 48213,
  "body_encoding": "utf-8",
  "truncated": false,
  "content_type": "text/html; charset=utf-8",
  "elapsed_ms": 1840,
  "blocked": false,
  "block_reason": null,
  "redirects": []
}

Two fields are easy to miss:

  • body_encoding is utf-8 or base64. Binary responses, and text we could not decode, come back base64. Check it before parsing.
  • truncated means the body hit the size cap and was cut. body_length is the byte length after truncation.

When ok is false, the shape is different — see Errors.