Run jobs and batches

Use a job when a request may take longer than your client can stay connected.
Use a batch when you have multiple inputs for the same capability.

Queue a job with the CLI

Set up the CLI first. This example also uses jq to
capture the returned job ID.

Add --detach to return a job ID immediately. Do not combine --detach with
--output.

JOB_RESPONSE="$(npx --yes @scrappycoco/cli@latest run web.crawl_site \
  --input '{"url":"https://example.com"}' \
  --detach --json)"
JOB_ID="$(printf '%s' "$JOB_RESPONSE" | jq -r '.job_id')"
npx --yes @scrappycoco/cli@latest jobs wait "$JOB_ID" \
  --output crawl.jsonl --format jsonl --json

Use jobs get "$JOB_ID" for one status check and jobs cancel "$JOB_ID" to
request cancellation. doctor --json diagnoses version, authentication,
API/catalog reachability, and skill state when setup is unclear.

Process several inputs in a batch

Batch execution prefers the selected provider's native bulk endpoint. Inspect
batch_mode, native_batch_max_size, and batch_reason in the live catalog.
Some endpoints execute individual requests; these explicitly report
batch_mode: individual. A provider's bulk support is capability-specific.

For a search batch, save this request as search-batch.json:

{
  "source": "web",
  "capability": "search_web",
  "input": {"queries": ["dentists in Boston", "dentists in Cambridge"], "country": "US"},
  "providers": ["cloro_google"],
  "limit": 10
}

scrappycoco run web.search_web --file search-batch.json --detach --json

Copy the returned job_id, then replace JOB_ID below:

scrappycoco jobs wait JOB_ID --output results.json --json

For any capability, input: {"items": [{...}, {...}]} accepts 1–500 independent
capability inputs. Each item must pass the inspected input schema; nested batches
are rejected. Existing extraction input.urls remains supported. limit applies
per input. concurrency bounds individual calls; native providers manage their
own batch concurrency. The REST and MCP interfaces accept the same input objects.

Responses include ordered batch_items, per-attempt batch_index and
batch_mode, and upstream batch IDs where available. Inspect all item statuses
and candidate evidence. Native batch usage is counted once; allocated shares are
identified when the upstream reports only an aggregate cost.

--retry-failed RUN_ID reuses only the failed inputs and exact original
configuration. It rejects resubmission when upstream tasks may still be running
or submission outcome is unknown; reconcile those task IDs first. Successful
items are not included in the retry. Cancellation does not guarantee that
already-submitted upstream work stops or avoids charges.

REST and MCP

For REST, submit the execution body to POST /api/v1/scrapers/jobs, then
read GET /api/v1/jobs/{job_id} until it completes. The
REST guide has copy-paste examples.
For MCP, call submit_scraper_job with the capability ID, input, providers,
options, and a unique idempotency_key. Read the returned job with get_job;
its optional wait_seconds lets you wait up to 60 seconds per call. Use
cancel_job to request cancellation.

Save your results

Job result bodies are available for 30 days after completion. Check
result_expires_at and download results you need to keep. Reading an existing
job does not run its providers again. See Errors and safe retries
for idempotency and result-expiry behavior.