Reusable website tools
Create a public, read-only website tool from a URL and task, then call its saved version with new inputs.
A website tool combines a saved extraction recipe with a parameterized operation on a public website. Submit a URL and a task description once; the service inspects the source, builds and validates a version, and returns structured data with the tool's input and output schemas. Subsequent calls use that version with fresh input values.
The normal execution path for an unchanged, supported website does not call an LLM again. A confirmed extraction failure can enter a bounded repair flow. Browser acquisition, page traffic, validation, and repair still have their normal costs; a repeat is not necessarily free.
This feature requires website tools and AI recipes to be enabled on the API deployment. An unavailable feature is an API error, not a reason to keep resubmitting the request.
Choose the request mode
| Request property | Use |
|---|---|
extraction | Extract an explicit field list or use a conventional extraction recipe. |
task | Create a parameterized website tool from a URL and task description. |
tool | Run a saved website tool with new inputs, optionally pinning its version. |
These properties are mutually exclusive. Omit all three for an ordinary page request. Website tools use individual requests, not Data API batches.
The examples below use the Data API base URL https://api.runharvester.com/api/v1/data. Authenticate with a Data API key having scrape:write for submissions and cancellation, and requests:read for inspection, results, and recipe discovery. See authentication.
1. Create a tool
Describe the records, fields, input parameters, and selection rules. example_input supplies initial values, not a fixed filter baked into the extraction program.
curl -X POST 'https://api.runharvester.com/api/v1/data/requests' \
-H "X-API-Key: $RUNHARVESTER_API_KEY" \
-H 'Content-Type: application/json' \
-H 'Idempotency-Key: hockey-tool-create-001' \
--data '{
"url": "https://www.scrapethissite.com/pages/forms/?q=Boston",
"mode": "http",
"proxy": "off",
"task": {
"description": "Read the public hockey table using its GET search. Expose a required string parameter q. Return all matching rows with name, year, wins and losses; year, wins and losses are integers. Keep every year for each team and follow observed pagination.",
"example_input": { "q": "Boston" }
}
}'202 Accepted means the asynchronous request was queued. Keep its id and the original request body and idempotency key. The submission also carries an extraction object identifying the draft recipe and run; a draft is not yet a callable, validated tool.
task.description is required and accepts 1–12,000 characters after trimming. example_input defaults to {} and supports at most 32 named primitive values: string, finite number, or boolean. Input strings are limited to 4,096 characters. Objects, arrays, and null are not tool input values. The generated input_schema is the authority for later calls.
2. Wait for the whole task
curl 'https://api.runharvester.com/api/v1/data/requests/<request-id>' \
-H "X-API-Key: $RUNHARVESTER_API_KEY"Poll with a finite application deadline and stop on completed, failed, or cancelled. An acquired page can be complete while extraction is still working. Inspect the request's overall status and its extraction state; do not treat page_status: "completed" as task completion.
A completed tool request includes metadata under extraction.tool:
{
"recipe_id": "550e8400-e29b-41d4-a716-446655440000",
"version_id": "550e8400-e29b-41d4-a716-446655440001",
"site_origin": "https://www.scrapethissite.com",
"input_schema": {
"type": "object",
"additionalProperties": false,
"properties": {
"q": { "type": "string", "maxLength": 4096 }
},
"required": ["q"]
},
"output_schema": {
"type": "object",
"properties": { "items": { "type": "array" } }
}
}This is an illustrative metadata subset, not a promise that every tool uses q or returns items. Use the schemas actually returned by your tool. Save both identities: recipe_id identifies the tool, while version_id identifies its immutable program and contract.
3. Retrieve structured data
After completion:
curl 'https://api.runharvester.com/api/v1/data/requests/<request-id>/result?format=data' \
-H "X-API-Key: $RUNHARVESTER_API_KEY"The format=data response contains id, format, data, result_meta, extraction, and expires_at. data is the structured result; extraction carries the tool metadata, public errors, coverage, and usage without repeating the payload. Other explicit formats return their selected page content instead. Omit format to receive the default HTML result and structured data together. Do not use page-content presence to determine whether extraction succeeded; inspect the request and extraction status.
Completion and coverage are different
{
"pages": 3,
"complete": true,
"stop_reason": "source_exhausted"
}extraction.coverage.stop_reason | Meaning |
|---|---|
source_exhausted | The supported operation reached the confirmed end of its source. |
page_limit | The operation stopped at its page bound; more source data may exist. |
result_limit | The result bound was reached; the output may be truncated. |
no_progress | Repeated or non-progressing source pages stopped the traversal. |
null | A terminal coverage reason is not available yet. |
Use coverage.complete, not HTTP 200 or status: "completed", to decide whether the selected source was exhausted. A successful request can return a bounded, incomplete result. Coverage describes this operation, not every record on the entire website.
A source search may be broader than the requested output filter. For example, a site may match product descriptions while the task asks for matching product names. A tool can apply explicit structured result filters after extracting each fresh source page. Source pagination still advances through pages with no selected records; an empty selected result is not proof that the source ended. These internal filters are generated and saved by the service, not passed as an extra task request property.
4. Call the saved version
Replace the example UUIDs with the identities returned by the completed creation request:
curl -X POST 'https://api.runharvester.com/api/v1/data/requests' \
-H "X-API-Key: $RUNHARVESTER_API_KEY" \
-H 'Content-Type: application/json' \
-H 'Idempotency-Key: hockey-tool-chicago-001' \
--data '{
"url": "https://www.scrapethissite.com/pages/forms/",
"proxy": "off",
"tool": {
"recipe_id": "550e8400-e29b-41d4-a716-446655440000",
"version_id": "550e8400-e29b-41d4-a716-446655440001",
"input": { "q": "Chicago" }
}
}'The URL must remain within the tool's site origin. The saved operation builds the actual request path and query from the new inputs; changing only the supplied URL does not override its program. An omitted version_id resolves the active version at admission; that admitted run stays pinned if the active version changes later.
Missing required inputs, unknown inputs, incompatible input types, an unowned recipe/version, or a site-scope mismatch fail instead of silently modifying the contract. The saved page configuration can require browser acquisition even when the submitted transport preference was HTTP.
Discover previously created tools
curl 'https://api.runharvester.com/api/v1/data/recipes?limit=20&offset=0' \
-H "X-API-Key: $RUNHARVESTER_API_KEY"The page contains items, total, limit, offset, and has_more. Recipe entries with valid active website-tool metadata expose tool; ordinary recipes must not be assumed to be parameterized website tools. Only the authenticated tenant's recipes are returned.
Usage and cancellation
Website tasks are charged from actual usage across creation, page acquisition, execution, validation, and any permitted repair. max_credits_per_request does not cap task/tool operations; do not use that legacy scrape setting as a website-tool spending limit. Normal account authorization and sufficient balance are still required.
extraction.budget reports charged_credits, reserved_credits, max_credits, and remaining_credits. A null maximum or remaining amount means no task-wide monetary cap, not zero or a missing price. Do not add llm_charged_tokens to budget.charged_credits: the latter already aggregates the task's accounted usage.
Cancel active work with POST /api/v1/data/requests/{id}/cancel. Cancellation is asynchronous and cannot undo an already completed external operation. Do not assume a cancellation immediately produces zero charged credits; inspect the returned state and subsequent request metadata.
Failures and safe retries
Keep HTTP admission errors separate from asynchronous extraction.error and extraction.user_error. Unsupported sources, required input, unavailable features, page failures, and unresolved model accounting require different actions. Never present an invalid or failed candidate as a saved, usable tool.
If the POST response is lost, resend the same body with the same idempotency key to resolve the same submission. A new key can create new work and new charges.
The generic /requests/{id}/retry path does not retry website tasks/tools: it returns WEBSITE_TOOL_RETRY_REQUIRES_NEW_REQUEST. To intentionally retry a terminal failure, submit a new task or tool request with a new idempotency key. Reuse an existing validated tool when appropriate; do not regenerate it just because a transport request failed. An unknown provider outcome requiring reconciliation is not permission for an automatic new paid attempt.
Only public read-only catalog/search retrieval is supported here. Do not supply credentials, non-empty cookies, headers, or browser_actions. Login, writes, arbitrary JavaScript, and universal website compatibility are not part of this contract. For authenticated or write workflows, see Browser Workflows.
SDK and MCP
The Node.js SDK uses the same postDataRequests operation for both modes. Pass task or tool in its body and an Idempotency-Key header, then use the existing request polling and result methods. Use a release whose generated types include these modes; do not replace them with unsafe casts to an older SDK.
With MCP, use data_create_request with body.task or body.tool and retain its idempotency_key. Inspect it with data_get_request, retrieve its structured output with data_get_result using format: "data", and discover saved metadata through data_list_recipes with detail: "full" when you need the input/output schemas. There is no new per-website MCP tool registration step. Keep the same input, idempotency, coverage, and cost rules as REST.
Last updated on