Control data freshness
Request fresh content or allow provider caches using explicit options.
Choose freshness based on your task: a recent price check may need a new fetch,
while a summary of a stable page may allow cached content.
Cache freshness is caller-controlled through each provider's native options.
Inspect the live capability schema before running: these controls apply through
REST, MCP, CLI, and supported native batches.
| Provider routes | Options and units |
|---|---|
| Firecrawl content, structured/typed extraction, crawl, monitoring, screenshot | max_age_ms (alias maxAge), milliseconds; 0 requests fresh content. Existing content/extraction/crawl/monitor routes default to 0; screenshots leave the upstream default when omitted. |
| Firecrawl search | Opt in to scraping result pages with scrape_options (alias scrapeOptions), containing max_age_ms / maxAge and optional formats. This controls result-page freshness, not the search index, and scraping adds provider usage. |
| Firecrawl map | ignore_cache (alias ignoreCache): true bypasses the sitemap cache. |
| Exa search, content, structured extraction, crawl, monitoring | max_age_hours (alias maxAgeHours), hours, up to 720; 0 requests fresh content, -1 requests cached content only. Search applies it to returned page content. Defaults: content/crawl 24, fresh-content/monitor 1, structured extraction 0; search leaves the upstream policy when omitted. |
| SerpApi search | no_cache: true bypasses its fixed one-hour cache; default false. |
| Scrapfly content, structured extraction, crawl, screenshot | cache enables reuse (default false), cache_ttl sets seconds (0–604800, default 86400), and cache_clear requests refresh. Cached JavaScript-rendered responses may omit screenshots; request fresh output when a screenshot is required. |
| ScrapeCreators routes exposing cached options | cache_max_age: 1d, 3d, 7d, 14d, or 30d. |
Example: allow content cached within the last hour
For web.extract_content with firecrawl_scrape, add these provider options:
{"firecrawl_scrape":{"max_age_ms":3600000}}Put that object under provider_options or pass it to --provider-options.
The unit is milliseconds: 1 means one millisecond, not one hour or one day.
These settings control provider caches, separately from Scrappycoco's 30-day
job-result retention. They do not guarantee a cache hit, a billing discount, or
cheaper extraction after changing the schema. Publication-date filters are not
cache controls. Providers without a documented control on the integrated route
do not accept a generic maxAge; unsupported options return a validation error.
Updated about 1 month ago