Extract content from URLs

Extract clean text, summaries or JSON fields from any URL with the Valyu Contents API. Batch up to 50 URLs, run async jobs and pay only for pages that succeed.

Already have a URL? Contents returns its readable text, an optional summary or fields extracted into JSON. Use Search first when you need to find the pages. You only pay for URLs that succeed.

Extract a page

Python

from valyu import Valyu

valyu = Valyu()  # reads VALYU_API_KEY from env

data = valyu.contents(
    urls=["https://en.wikipedia.org/wiki/Artificial_intelligence"],
    response_length="medium",
    extract_effort="auto",
)
for result in data.results:
    if result.status == "success":
        print(str(result.content)[:500])
    else:
        print(result.url, result.error)

Returns clean markdown for each URL. Pass up to 10 URLs synchronously, or up to 50 with async mode.

Core options

Option	Values	Default
response_length	short (25k), medium (50k), large (100k), max, or a char count (1k-1M)	short
extract_effort	normal (HTTP extraction), high (browser rendering), auto	auto
summary	false, true, an instruction string, or a JSON schema	false
screenshot	true to capture a page screenshot (not for PDFs)	false

Point-in-time content

To read a page as it was at a past date, set start_date / end_date with historical_cache: true (Python start_date, end_date, historical_cache; TypeScript startDate, endDate, historicalCache). Dates accept YYYY, YYYY-MM-DD or an ISO-8601 datetime. Contents then serves only a snapshot captured inside the window: with historical_cache on, Contents always behaves as historical_cache_strict: "only", whatever value you pass, and never falls back to a live crawl. A URL with no snapshot in the window fails rather than returning newer content.

response = valyu.contents(
    ["https://example.com/pricing"],
    end_date="2025-06-30",
    historical_cache=True,
)

Academic papers

When you pass an academic paper URL or DOI, Valyu serves it from our pre-processed academic index rather than crawling the live page. You get clean markdown with preserved figures, equations, tables, and section structure, plus inline pre-signed image URLs - the things lost when a JavaScript-heavy or PDF paper is scraped live. This happens automatically; just pass the URL.

Source	Example URL / id	Plan for index access
arXiv	https://arxiv.org/pdf/2506.05440 (or /abs/, /html/)	tier_0 (pay-as-you-go)
PubMed Central	https://pmc.ncbi.nlm.nih.gov/articles/PMC10584482/	tier_0
bioRxiv	https://www.biorxiv.org/content/10.1101/2020.07.28.225581	tier_1
medRxiv	https://www.medrxiv.org/content/10.1101/...	tier_1
ChemRxiv	https://chemrxiv.org/doi/full/10.26434/chemrxiv-...	tier_2

If your plan covers the source, you get the processed markdown; if the specific paper isn’t indexed yet, it falls back to live extraction. If your plan doesn’t cover the source, the URL is live-crawled when the page is publicly accessible; extraction can still fail.

Academic results set source to the dataset id (e.g. valyu/valyu-arxiv) and include doi, authors, and citation_count. Figures are returned in image_url as a filename→URL map and referenced inline in the markdown.

AI summaries and structured extraction

The summary field controls AI post-processing:

• false - raw markdown, fastest and cheapest (no AI)
• true - a basic AI summary
• string - custom instructions, e.g. "Summarise the methodology and key findings in 2-3 paragraphs"
• object - a JSON schema for structured extraction

# Structured extraction
data = valyu.contents(
    urls=["https://example.com/product-page"],
    summary={
        "type": "object",
        "properties": {
            "product_name": {"type": "string"},
            "price": {"type": "number", "description": "Price in USD"},
            "availability": {
                "type": "string",
                "enum": ["in_stock", "out_of_stock", "preorder"],
            },
        },
        "required": ["product_name", "price"],
    },
)
print(data.results[0].content)  # returns a JSON object

Summary examples (all four modes)

Python

# false - raw markdown, no AI
valyu.contents(urls=["https://example.com/article"], summary=False)

# true - basic AI summary
valyu.contents(urls=["https://example.com/article"], summary=True)

# string - custom instructions
valyu.contents(
    urls=["https://example.com/research-paper"],
    summary="Summarise the methodology, key findings, and applications in 2-3 paragraphs",
)

# object - JSON schema extraction
valyu.contents(
    urls=["https://example.com/product-page"],
    summary={
        "type": "object",
        "properties": {
            "product_name": {"type": "string"},
            "price": {"type": "number"},
            "features": {"type": "array", "items": {"type": "string"}, "maxItems": 5},
        },
        "required": ["product_name", "price"],
    },
)

JSON schema reference and tips

Use any valid JSON Schema. Limits: 5,000 characters, 3 levels deep, 20 properties per object.

Supported types: string, number/integer, boolean, array, object.

Tips:

• Use clear description fields to guide extraction.
• Use enum for consistent categorisation.
• Keep schemas shallow and mark essential fields required.

Screenshot capture

Python

data = valyu.contents(
    urls=["https://example.com/article"],
    extract_effort="auto",
    screenshot=True,
)
print(data.results[0].screenshot_url)

Screenshots are captured during page rendering and returned as pre-signed URLs. PDF files do not support screenshots.

Async processing

For 11-50 URLs (required above 10) or non-blocking workflows, use async mode. Submit URLs, get a job_id immediately, then poll or receive results via webhook. Async also raises the per-URL timeout to 120s (vs 25s for sync).

job = valyu.contents(
    urls=["https://example.com/page1", "https://example.com/page2"],  # up to 50
    async_mode=True,
    webhook_url="https://your-app.com/webhooks/valyu",  # optional
)
print(job.job_id)

# Block until complete (SDK handles polling)
result = valyu.wait_for_contents_job(job.job_id, poll_interval=5, max_wait_time=3600)
for r in result.results or []:
    if r.status == "success":
        print(r.title, r.length)
    else:
        print(r.url, r.error)
print(f"Total cost: {result.actual_cost_dollars}")

Async mode is required above 10 URLs. For 1-10 URLs it’s optional, for non-blocking workflows.

Polling, job lifecycle, and status fields

The initial submit returns HTTP 202 with a job_id, poll_url, and a one-time webhook_secret (store it immediately - you cannot retrieve it later).

Jobs move through these statuses:

Status	Meaning
pending	Created, not yet started
processing	URLs running in batches of 5
completed	All URLs succeeded
partial	Finished with some failures
failed	All URLs failed

To poll manually:

Python

import time

while True:
    status = valyu.get_contents_job(job.job_id)
    if status.status in ("completed", "partial", "failed"):
        break
    time.sleep(2)

if status.status in ("completed", "partial"):
    for result in status.results or []:
        if result.status == "success":
            print(result.title, result.length)
        else:
            print(result.url, result.error)

Status response fields include: job_id, status, urls_total, urls_processed, urls_failed, created_at, updated_at, current_batch / total_batches (while processing), results and actual_cost_dollars (when completed/partial), and error (when partial/failed).

Webhooks and signature verification

When you provide a webhook_url, Valyu POSTs to it when the job finishes (completed, partial, or failed). Headers: Content-Type: application/json, User-Agent: Valyu-Contents/1.0, X-Webhook-Signature: sha256={hex}, X-Webhook-Timestamp (Unix seconds).

Retries: up to 5 attempts with exponential backoff (1s, 2s, 4s, 8s), no retry on 4xx, 5s connect / 15s read timeout.

The payload is signed with HMAC-SHA256 over "{timestamp}.{json_payload}". Verify the original request body before parsing or reserializing JSON. This example checks the signature; also check the timestamp against your accepted replay window and deduplicate processing by job ID:

Python

import hmac, hashlib

def verify_webhook(payload: bytes, signature_header: str, timestamp_header: str, webhook_secret: str) -> bool:
    signed = f"{timestamp_header}.{payload.decode('utf-8')}"
    expected = hmac.new(webhook_secret.encode(), signed.encode(), hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, signature_header.removeprefix("sha256="))

The TypeScript SDK exports verifyContentsWebhookSignature() which handles this for you.

Limits and pricing

Detail	Value
Max URLs (sync / async)	10 / 50
Batch size	5 URLs
Timeout per URL (sync / async)	25s / 120s
Base pricing	0.2 credits per successfully extracted URL
AI features (summary/schema)	+0.2 credits per URL, 0.4 credits in total
Job expiry (TTL)	7 days

Failed and blocked URLs are free. For example, 8 successful URLs use 1.6 credits, or 3.2 with summary set. total_cost_dollars (sync) and actual_cost_dollars (async) are the amounts charged. Each result’s price is the base extraction price only and excludes the AI add-on.

max_price_dollars caps the charge for a request. If the worst case exceeds it, the request is rejected with 400 before any work starts; if the actual cost exceeds it, you get 402.

Response format

Illustrative response:

{
  "success": true,
  "tx_id": "tx_12345678-1234-1234-1234-123456789abc",
  "results": [
    {
      "status": "success",
      "title": "AI Breakthrough in Natural Language Processing",
      "url": "https://example.com/article?utm_source=valyu",
      "content": "# AI Breakthrough...\n\nPage content in markdown...",
      "source": "web",
      "price": 0.00125,
      "length": 12840,
      "data_type": "unstructured"
    }
  ],
  "urls_requested": 1,
  "urls_processed": 1,
  "urls_failed": 0,
  "total_cost_dollars": 0.00125,
  "total_characters": 12840
}

With summary set, content is the AI summary (string) or extracted JSON (object), and each result adds summary_success.

Result fields

Field	Description
status	"success" or "failed" for this URL
error	Human-readable failure reason (failed results only)
title	Extracted page title
url	Original URL
content	Extracted content (markdown or JSON)
description	Page meta description (paper abstract for academic sources)
source	"web", or the dataset id for academic papers (e.g. valyu/valyu-arxiv)
price	Base extraction price for this URL, excluding the AI add-on. Use total_cost_dollars for the charge
length	Character count
data_type	"unstructured" or "structured"
summary_success	Whether AI processing succeeded (only when summary is used)
image_url	Extracted image URLs (figures, for academic papers)
screenshot_url	Page screenshot URL (omitted unless screenshot=true)
doi, authors, citation_count	Academic sources only

Error handling

Each result carries a status of success or failed. Failed results include a descriptive error (page unavailable, required sign-in, blocked, timed out) - inspect status before reading the content.

HTTP status	Meaning
200	All URLs extracted successfully
206	Partial - some succeeded, some failed (check urls_failed and per-result status)
422	Every URL failed - see the top-level error

The following uses a raw HTTP response, rather than the SDK response model.

if response.status_code == 206:
    data = response.json()
    failed = [r for r in data["results"] if r.get("status") == "failed"]
    for r in failed:
        print(f"{r['url']} failed: {r['error']}")
elif response.status_code == 422:
    error_message = response.json()["error"]

Next steps

API reference

Complete parameter documentation

Python SDK

Python integration

TypeScript SDK

TypeScript integration

Integrations

LangChain, LlamaIndex, and more