Errors and safe retries

REST errors use a standard HTTP status and a JSON detail field.

Safe retries

Use a unique Idempotency-Key for every logical billable POST. Reuse the value
only when retrying the identical request body. Reusing it with different input
returns 409.

Honor an API Retry-After header before retrying temporary 429 and 503
responses. When no delay is available, use bounded exponential backoff and jitter.
Do not blindly retry validation, authentication, billing, or authorization
errors. Inspect provider-level attempts and next_action before deciding how
to recover from a partial execution.

An external provider can fail inside a completed Scrappycoco request. Inspect
attempts[].status, attempts[].provider_http_status, and attempts[].error;
do not assume the outer API status describes every provider attempt. A Run with
no records and only failed or unavailable provider attempts returns
status: "failed".

Provider attempts also expose error_code, retryable, and
retry_after_seconds. Retryability describes a transport condition, not a
promise that another call will succeed. null means unknown; a missing delay
does not mean retry immediately. Delay values come from the upstream
Retry-After header (seconds or HTTP date), not a predicted congestion window.
Use these fields to decide when to retry. provider identifies the affected configuration;
provider_id identifies its upstream when using named comparison candidates.

For failed structured extraction, attempts[].native_response preserves the
provider payload, including warnings when supplied. A missing result does not
prove that the provider incurred no usage. Inspect the billing evidence before
deciding to submit a new billable request.

LinkedIn cursors are scoped to the provider, capability, input and exact options
that produced them. Reuse that configuration when continuing a page. A cursor
that matches none of the selected configurations returns HTTP 422 before a
job or billing reservation is created, including in batches and saved Discovery
runs. If an already queued request or a non-matching fallback candidate reaches
execution, its invalid_cursor attempt records exact zero cost and makes no
upstream request. Actual upstream failures can still incur provider usage.

Provider fallback

An explicit providers array is the complete ordered execution list. For example,
["cloro_google"] has no fallback. Run tries a second selected provider after
an error or empty result, and stops when a provider returns records. It does not
test task-specific relevance or required-field completeness. Inspect attempts
and compare the exact provider options before selecting a backup.

In the CLI, repeat --provider for an ordered fallback list. If no providers are
selected, Run uses the first two available providers in catalog order. Availability
alone does not make providers equivalent for Google rankings or local-pack fields.
Failures before provider execution, such as job initialization or admission errors,
occur outside this fallback loop.

Idempotency window and saved jobs

The idempotency window is 24 hours from the original submission.
This is distinct from
job/result storage retention. Reading an existing job with GET /jobs/{id}
does not execute its provider requests again.

Across all providers, job result bodies are available for 30 days after completion, including
native outputs and comparison evidence. result_expires_at reports the deadline.
After expiry, job status remains available, result is null, and
result_expired is true. Billing, usage, and request configuration records
remain. Download results within the delivery window for long-term reuse.

Within both windows, an identical POST and key replays the stored response.
If an idempotency key remains reserved after result delivery expires (for example,
a cancelled job), replay returns 410 instead of returning an empty result. After a completed or failed key expires, reusing it can
create a new execution and new charges. Replays do not extend the window.
Pending/running requests are protected from expiry while active. Keep job IDs
and export results for audit workflows; do not use old POST requests as a
permanent result cache. To retry a completed execution whose provider attempt
failed, inspect the evidence and submit a new logical request with a new key.

Input limits and structured output

limit is a positive top-level upstream count or collection target, default 10.
It has no universal maximum; provider endpoint limits still apply. This also
applies to web.search_web and web.crawl_site. Complete returned provider pages
are preserved, so normalized records may exceed the requested count. The CLI
--limit flag maps to this field. Putting it inside input produces a validation
error. For example:

{
  "source": "web",
  "capability": "crawl_site",
  "input": {"url": "https://example.com", "max_depth": 2},
  "limit": 25
}

Scrappycoco forwards the structured extraction schema without an eight-property
cap. Provider extraction can still fail; report the request ID, URL, prompt,
schema, and native response when diagnosing schema-dependent failures.
Native string values such as "null" or "none" are preserved. If absent
fields should be JSON null, describe that in the prompt and allow null in the
schema; the calling client should validate the returned values.

CLI exit codes