Quickstart
Make three API calls to submit a URL, check its status, and retrieve the result.
RunHarvester fetches pages asynchronously: submit a URL, poll the returned request ID, then retrieve the result. This walkthrough uses Bash, cURL, and Python 3. On Windows, use WSL or Git Bash, or follow the Python guide.
1. Set up your API key
Create an account at app.runharvester.com, verify your email, and create a Data API key with scrape:write and requests:read scopes. Keep the key on a trusted machine or server.
export RUNHARVESTER_API_KEY='rh_live_replace_me'
export RUNHARVESTER_IDEMPOTENCY_KEY="$(python3 -c 'import uuid; print(uuid.uuid4())')"Generate the idempotency key once for this submission. If the POST times out, reuse the same key and body; generate a new key only for a new logical request.
2. Submit a URL
curl --fail-with-body --silent --show-error --max-time 20 \
'https://api.runharvester.com/api/v1/data/requests' \
--header "X-API-Key: $RUNHARVESTER_API_KEY" \
--header "Idempotency-Key: $RUNHARVESTER_IDEMPOTENCY_KEY" \
--header 'Content-Type: application/json' \
--data '{"url":"https://example.com","mode":"auto"}'An accepted submission returns HTTP 202 and an id. Save that ID. Acceptance means the job exists; it does not mean the page is ready. A cache hit may already be completed.
3. Check the request status
Replace the placeholder with the id from the submission response:
export RUNHARVESTER_REQUEST_ID='replace_with_request_id'
curl --fail-with-body --silent --show-error --max-time 20 \
"https://api.runharvester.com/api/v1/data/requests/$RUNHARVESTER_REQUEST_ID" \
--header "X-API-Key: $RUNHARVESTER_API_KEY"Read status in the JSON response:
| Status | Next step |
|---|---|
pending, running, solving_captcha | Wait at least a second, then repeat the status request. |
completed | Retrieve the result below. |
failed, cancelled | Stop polling and inspect error or user_error. See Errors. |
Stop this walkthrough if the job has not finished after two minutes. Keep its ID: stopping your local poll does not cancel server-side work. For automated polling with a deadline, use the complete cURL script.
4. Retrieve the result
Only run this after the status is completed:
curl --fail-with-body --silent --show-error --max-time 20 \
"https://api.runharvester.com/api/v1/data/requests/$RUNHARVESTER_REQUEST_ID/result" \
--header "X-API-Key: $RUNHARVESTER_API_KEY"The response contains the acquired page result; structured extraction adds typed data when requested. Validate the fields your application needs before storing them.
409 RESULT_NOT_READY means you should continue polling. 410 RESULT_EXPIRED means the retained result is no longer available; submit a new request if you still need it.
Next steps
- Automate the lifecycle with cURL, TypeScript / JavaScript, or Python.
- Learn retry and idempotency rules before adding automatic retries.
- Review billing and limits before processing many URLs.
Last updated on