Rust SDK: Contents

Extract clean text, AI summaries or structured JSON from URLs in Rust with the Valyu SDK, including async jobs for up to 50 URLs per request.

The Rust SDK is in alpha. Interfaces may change between releases; check the crate version before upgrading.

Extract clean, structured content from web pages, with optional AI summarization and structured data extraction.

Basic usage

use valyu::{ValyuClient, ContentsRequest};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = ValyuClient::new(std::env::var("VALYU_API_KEY")?);

    let request = ContentsRequest::new(vec![
        "https://en.wikipedia.org/wiki/Machine_learning".to_string(),
    ]);

    let response = client.contents(&request).await?;

    println!("Processed {}/{} URLs",
        response.urls_processed.unwrap_or(0),
        response.urls_requested.unwrap_or(0));

    if let Some(results) = &response.results {
        for result in results {
            println!("{} ({} chars)",
                result.title.as_deref().unwrap_or("Untitled"),
                result.characters.unwrap_or(0));
        }
    }

    Ok(())
}

Check response.success and response.urls_failed before using the returned content, and handle missing content values. Extraction depends on page access and layout; a subscription does not guarantee access to an arbitrary paywalled URL.

Common patterns

use serde_json::json;
use valyu::ContentsRequest;

// Auto AI summary
let req = ContentsRequest::new(urls.clone())
    .with_summary(true)
    .with_response_length("medium");

// Custom summary instruction
let req = ContentsRequest::new(urls.clone())
    .with_summary_instructions("Summarize the main trends in 3 bullet points")
    .with_extract_effort("high");

// Structured extraction - pass a JSON schema, get back structured fields
let schema = json!({
    "type": "object",
    "properties": {
        "company_name": {"type": "string"},
        "industry": {"type": "string"},
        "founded_year": {"type": "number"}
    },
    "required": ["company_name"]
});
let req = ContentsRequest::new(vec!["https://en.wikipedia.org/wiki/OpenAI".to_string()])
    .with_summary_schema(schema)
    .with_extract_effort("high");

let response = client.contents(&req).await?;

Use with_extract_effort("high") for JS-heavy pages, and with_response_length ("short" 25k, "medium" 50k, "large" 100k, "max") or with_custom_response_length(i32) to control content per URL.

For arXiv, PubMed Central, bioRxiv, medRxiv, and ChemRxiv papers, Valyu serves clean processed markdown (with figures and equations) from its academic index when your plan covers the source - otherwise it uses the live crawler. Pass the paper URL (a /pdf/ arXiv link or a DOI works best) or bare id. See Academic Papers.

Reference

Builder methods

urls (Vec<String>, required) - URLs to process (max 10).

Method	Type	Description	Default
with_response_length()	impl Into<String>	"short" (25k), "medium" (50k), "large" (100k), "max"	"short"
with_custom_response_length()	i32	Custom character limit (1K-1M)	None
with_extract_effort()	impl Into<String>	"normal", "high", or "auto"	"auto"
with_summary()	bool	Enable default summarization	false
with_summary_instructions()	impl Into<String>	Custom summary instructions	None
with_summary_schema()	serde_json::Value	JSON schema for structured extraction	None
with_max_price_dollars()	f64	Cap on the cost of the whole request. See pricing	None

Response format

pub struct ContentsResponse {
    pub success: bool,
    pub error: Option<String>,
    pub tx_id: Option<String>,
    pub urls_requested: Option<i32>,
    pub urls_processed: Option<i32>,
    pub urls_failed: Option<i32>,
    pub results: Option<Vec<ContentResult>>,
    pub total_cost_dollars: Option<f64>,
    pub total_characters: Option<i32>,
}

pub struct ContentResult {
    pub url: Option<String>,
    pub title: Option<String>,
    pub content: Option<serde_json::Value>,
    pub description: Option<String>,
    pub publication_date: Option<String>,
    pub images: Option<Vec<String>>,
    pub cost_dollars: Option<f64>,
    pub characters: Option<i32>,
}

Error handling

use valyu::{ContentsRequest, ValyuError};

let request = ContentsRequest::new(vec!["https://example.com".to_string()]);

match client.contents(&request).await {
    Ok(response) => {
        if !response.success {
            eprintln!("Extraction failed: {:?}", response.error);
            return;
        }
        if let Some(failed) = response.urls_failed {
            if failed > 0 {
                println!("{} URLs failed", failed);
            }
        }
        if let Some(results) = &response.results {
            for result in results {
                println!("{} ({} chars)",
                    result.title.as_deref().unwrap_or("Untitled"),
                    result.characters.unwrap_or(0));
            }
        }
    }
    Err(ValyuError::InvalidApiKey) => eprintln!("Invalid API key"),
    Err(ValyuError::RateLimitExceeded) => eprintln!("Rate limit exceeded"),
    Err(e) => eprintln!("Error: {}", e),
}