Best Web Search and Scraping APIs for AI Agents in 2026

Pick the API that matches the job your agent actually does. For questions that need fresh answers, Exa, Perplexity and Parallel are the three that keep landing at the top of the public boards. For turning a URL into clean markdown, Firecrawl and Jina are the defaults. For a Google-shaped result page, use SerpAPI; for a hard-to-scrape site, use Bright Data or an Apify Actor. Brave is the choice when you want an independent index and clear data terms, and Linkup is a lean option that bills per call.

The catch is that every vendor publishes a benchmark where it wins, prices are changing month to month, and the word “search” covers four different products. This page fixes the vocabulary, lines up the prices as of Nov 11, 2026, labels who ran each benchmark, and gives you a short test plan so you can decide with your own queries. If you are still choosing how agents reach the open web at all, start with browser infrastructure for AI agents.

Official Firecrawl intro video (YouTube, title verified through YouTube oEmbed): the search, scrape and interact flow that most vendors on this list now copy in some form.

Quick Picks

These are editorial calls, not paid placements. Each pick links to the section that explains the trade-offs, and every price is dated as of Nov 11, 2026.

Master Comparison

One row per vendor. “LLM-ready markdown” means the API returns page content as markdown or clean text without you writing a parser. Every cell was read from the vendor's pricing, docs or GitHub page on Nov 11, 2026. Where a vendor does not publish a number, the cell says so rather than guessing.

Open source is mixed. Brave's MCP server and Jina's Reader code are public, Firecrawl and Apify have open-source components, and the search indexes behind Exa, Tavily, Parallel and Linkup are proprietary hosted services. We do not mark a vendor open source here unless its repository or page said so.

Search vs Crawl vs Extract

Four products hide behind the word search, and mixing them up is the most common reason teams overpay or under-deliver.

• SERP APIs (SerpAPI, Bright Data SERP) return the result page of an existing engine, parsed to JSON. You get what Google or Bing shows, including ads, knowledge panels and local packs.
• Index-backed search APIs (Brave, Exa, Parallel, Linkup, Tavily, Perplexity Search) run against their own or blended index and return ranked results with snippets, often with extracted highlights meant for a model's context window.
• Extract or reader APIs (Jina Reader, Firecrawl scrape, Tavily extract, Exa contents, Parallel extract, Linkup fetch) take a URL and return markdown or JSON, optionally rendering JavaScript.
• Crawlers and scraper marketplaces (Firecrawl crawl and map, Tavily crawl and map, Apify Actors, Bright Data scrapers) walk many pages or run site-specific extraction.

A typical grounding loop is one search call, then an extract on the best two or three hits. That means two prices matter, not one, and a vendor that bundles both in a single request (Firecrawl search with scraping, Exa contents, Linkup standard) can be cheaper than it looks, or dearer, depending on how many pages you actually read. Local documents are a separate problem: for PDFs and Office files see the Firecrawl AnyDoc guide.

Firecrawl

What it is. A web data API with search, scrape, crawl, map, an agent endpoint and an interact endpoint, all billed in credits. It is the one vendor on this list that treats extraction as the main product and search as the way in.

Pricing (as of Nov 11, 2026). Free is $0 with 1,000 credits, 2 concurrent requests and no card required. Hobby is $19 per month for 5,000 credits, Standard $99 for 100,000, Growth $399 for 500,000 and Scale $749 for 1,000,000, with annual billing discounts. Credit costs: scrape and crawl 1 credit per page (plus 4 for JSON format), search 2 credits per 10 results (more if you scrape the results), map 1 credit per call. The pricing page does not list an Extract price, and states pay-as-you-go is not available and credits do not roll over.

Rate limits. Free allows 10 requests per minute on scrape, map and search and 2 per minute on crawl. Standard allows 500 and 100, Growth 5,000 and 1,000, Scale 10,000 and 2,000.

Strengths. One credit system covers finding and reading pages, which keeps agent code short. The hosted MCP server supports OAuth sign-in and an API-key endpoint, and its docs say Search, Scrape and Parse work in a keyless mode within daily limits. In Openbenchmarks' search-only hard-retrieval test it completed 70.3 percent of tasks at a median $0.059, which is strong for the price.

Weaknesses. No pay-as-you-go, so low-volume or bursty use means paying for a plan tier. On multi-hop company discovery it scored an F1 of 30.4 (search only) against 45.4 for Exa deep, so it is not the best pure discovery engine. Its own SimpleQA claim of 94.7 percent is vendor-reported with no competitor numbers.

Best for. Agents that read whole pages, RAG ingestion pipelines, and teams that want search, scrape and crawl from one vendor. Also see the AnyDoc guide for local files.

Exa

What it is. An embeddings-first search engine built for programmatic use, with instant, fast, auto and deep search types, a contents endpoint, an answer endpoint, monitors, and an agent product for multi-step research.

Pricing (as of Nov 11, 2026). Signup gives $10 in credits that reset to $10 on the first of each month, no payment method required (up to 2,500 lite searches). Search is $4 per 1K for lite, $7 per 1K for fast and auto, with each result above 10 costing $1 per 1K. Deep search is $12 per 1K. Contents cost $1 per 1K pages per content type, answers $5 per 1K, monitors $2 per 1K. Agent calls are priced per effort level, from $0.012 (minimal) to $1.00 (xhigh). The Enterprise plan is $12,000 per year with Zero Data Retention and 25 QPS; the page lists no QPS for the self-serve tier.

Strengths. It tops several independent-style boards: 99.3 percent on Openbenchmarks' factual lookup (with the bias caveat above), 83.0 completion on hard retrieval with fetch, and the best multi-hop F1 (48.2). Latency is low, with instant search at 398 ms on that board and 0.74 s p50 in the cloro run.

Weaknesses. At $7 per 1K it costs more per request than Brave, Perplexity and Linkup, and cost per task grows on multi-hop work (about $0.65 to $0.72 per question in Openbenchmarks' multi-hop test). Rate limits for self-serve plans are not published.

Best for. Semantic discovery, company and people research, and agents that need good results on the first call. Exa has been shipping agent-oriented features quickly, and its own X posts show the direction:

Both posts are first-party announcements of Exa Connect (partner data such as ZoomInfo and Similarweb) and its arrival in the Exa MCP server. Partner providers bill separately from Exa.

Tavily

What it is. A search API aimed at LLM agents, with extract, crawl, map and a research endpoint.

Pricing (as of Nov 11, 2026). The Researcher plan is free with 1,000 credits and no card. Pay As You Go is $0.008 per credit. The Project plan shows $30 per month for 4,000 credits (the page text was garbled by a price slider, so confirm it in your dashboard). Credit costs from the docs: basic search 1 credit, advanced search 2, extract 1 credit per 5 successful URLs on basic and 2 on advanced, map 1 credit per 10 pages, crawl combines map plus extract (for 10 pages, 3 credits basic or 5 advanced), and research costs 4 to 110 credits on mini and 15 to 250 on pro. Failed extractions are not charged. It also advertises 100,000 free credits for startups and free student access.

Rate limits. Search is 100 requests per minute on the free plan, 1,000 on pay-as-you-go and the Project tier, and 1,500 on Growth and Scale. Crawl and map are 100 per minute, research 20.

Strengths. The five MCP tools (search, extract, crawl, map, research) cover the whole loop, and the remote server installs with one command in Claude Code. The free tier is generous for prototypes.

Weaknesses. In the independent and rival boards above it usually lands lower: 91.8 percent on cloro's SimpleQA, 60.0 completion on Openbenchmarks' hard retrieval with fetch, and the highest cost per task of the major vendors ($0.153). Advanced search at 2 credits is $16 per 1K on pay-as-you-go. Tavily's own blog claims first place on SealQA and SimpleQA, which we could not verify here.

Best for. Prototypes, LangChain-style agents that already ship a Tavily tool, and teams that value the free tier and the all-in-one MCP server more than peak accuracy.

Jina Reader and Search

What it is. Two simple endpoints: r.jina.ai/<url> returns a page as markdown, and s.jina.ai searches the web and returns content. There is also a remote MCP server with 12 tools, including PDF extraction, arXiv and SSRN search, and reranking helpers.

Pricing and limits (as of Nov 11, 2026). Reader works without a key. Every new API key includes 10M free tokens. Billing is token-based: Reader counts tokens on the output, and each Search request costs a fixed number of tokens starting at 10,000. Failed requests are not charged. The Reader limit is 20 RPM with no key, 500 RPM with a free or paid key and 5,000 RPM on premium; Search is blocked without a key, then 100 RPM free and paid and 1,000 RPM premium. The page lists an average Reader latency of 7.9 seconds and Search at 2.5 seconds. It shows no per-token dollar price in the text we retrieved. Third-party sites cite around $0.05 per 1M tokens, but they disagree with each other and one Oct 24, 2026 report claims the premium mode now costs 40 times more tokens, so verify before budgeting.

Strengths. The lowest friction of any tool here: a URL prefix and no signup. Good for prototyping and for single-page reads.

Weaknesses. Slower than the dedicated search engines, no crawl endpoint on the page we fetched, and licensing needs care: the page says ReaderLM-v2 and jina-vlm are CC-BY-NC 4.0, not open source, and commercial production use requires a license that Elastic has sold separately since August 10, 2026.

Best for. Quick page-to-markdown conversion, hobby agents and evaluation. For commercial systems, confirm the terms for the exact endpoint you use.

Brave Search API

What it is. Search backed by Brave's own independent index, with a plan for raw results and a plan for grounded answers. An LLM Context endpoint returns extracted, model-ready snippets.

Pricing and limits (as of Nov 11, 2026). The Search plan is $5 per 1,000 requests with $5 in free credits each month and 50 QPS. The Answers plan is $4 per 1,000 requests plus $5 per million input and output tokens, with $5 free monthly and 2 QPS. Enterprise is custom and includes full Zero Data Retention; the FAQ says storing results or using them for training requires a custom plan. The page's hero also says “up to $20 in free calls every month” while the plan cards say $5, and the page does not reconcile the two, so count on $5.

Strengths. An independent index means no dependence on Google or Bing, and Zero Data Retention is available. The MCP server is MIT-licensed in the brave GitHub organization with eight tools, including brave_llm_context and brave_summarizer. Brave LLM Context scored 94.0 percent at 601 ms on Openbenchmarks' factual board.

Weaknesses. It does not read pages for you, so you pair it with an extract tool. On harder tasks it trails: 38.0 completion on search-only hard retrieval and an F1 of 28.0 on multi-hop, versus 77.3 and 45.4 for the leaders.

Best for. Privacy-sensitive products, news and general web lookups, and teams that want a predictable $5 per 1K from an independent source.

Perplexity Search and Sonar

What it is. Perplexity now sells a raw Search API, an Agent API with metered tools, embeddings and a decisions API. Sonar, its older grounded-answer model family, is in transition.

Pricing and limits (as of Nov 11, 2026). Search API: $5.00 per 1K requests standard and $1.00 per 1K with search_type: "fast"; a request with up to five queries is one billing unit. The rate limit is 50 query units per second for every usage tier. Agent API tools are metered, for example web_search at $0.0025 per call ($0.001 fast), fetch_url at $0.0005, people_search and finance_search at $0.005. The docs state Sonar Chat Completions support ended on September 27, 2026 and map Sonar Pro to the Agent API's fast effort level. Legacy Sonar token and request fees did not appear on any first-party page we fetched, so we omit them.

Strengths. It is the best value on the hard-retrieval boards: 77.3 completion search-only at a median $0.059, and 46.6 F1 multi-hop with fetch. Parallel's BrowseComp run also shows Perplexity tied for the frontier-tier lead at 74 percent. The MCP server (npm @perplexity-ai/mcp-server, or hosted at api.perplexity.ai/mcp) exposes search, ask, research and reason tools.

Weaknesses. No free credits are listed on the pricing page we read, and the Sonar-to-Agent API migration is still rolling out gradually, so budget for change. Search only, with no crawl product.

Best for. Agents that want high-quality search at a low per-call price and are comfortable adopting the Agent API.

SerpAPI

What it is. A scraper-as-a-service that returns structured JSON for the result pages of Google, Bing, YouTube, eBay and many other engines.

Pricing and limits (as of Nov 11, 2026). Free: 250 searches per month at 50 per hour. Starter: $25 for 1,000 (200 per hour). Developer: $75 for 5,000 (1,000 per hour). Production: $150 for 15,000 (3,000 per hour). Searcher: $725 for 100,000 (20,000 per hour). The page lists 62 plans up to $98,325 for 50M searches, which works out to about $0.002 per search at the top. Per-search figures here are our arithmetic, not a listed price. Speed modes multiply the price by 2 or 4, and the free plan excludes Legal Shield and ZeroTrace mode.

Strengths. Exact fidelity to what Google shows, including local and shopping results. The hosted MCP server at mcp.serpapi.com provides a search tool across engines and an extract tool (scrape, map or crawl modes returning JSON or Markdown).

Weaknesses. Expensive at low volume ($15 to $25 per 1K), slow in cloro's run (8.20 s p50), and it often needed a second search, which pushed the effective cost to about $14.28 per 1,000 questions. On Openbenchmarks' multi-hop test a generic SERP API (a RapidAPI product, not SerpAPI) scored near zero, which shows raw result pages are not agent-shaped without extra work.

Best for. SEO, shopping, maps and local-intent agents where the exact SERP is the product.

Bright Data

What it is. A large web data platform: SERP API, unblocking proxies, a scraping browser, ready-made scrapers and datasets, with an MCP server that bundles them.

Pricing (as of Nov 11, 2026). The SERP API is $1.5 per 1K requests pay-as-you-go (you pay only for success), a $499 per month Scale plan with 380K requests and $1.3 per 1K beyond that, a free tier of 5K requests per month with no card, and a first-deposit match up to $500. Web Unlocker pricing was not on the page we read, and the pricing page states no rate limit.

Strengths. The cheapest SERP per request here, and the broadest anti-bot toolkit. The MCP server (30 tools) includes search_engine, scrape_as_markdown, batch variants, scraper-API tools, 14 browser-automation tools and dataset search. Its free tier of 5,000 requests per month renews on the 1st.

Weaknesses. Breadth means complexity, and unit prices across products are harder to compare than a single per-1K figure. It was not in the Parallel, cloro or Openbenchmarks tests we read, so we have no public accuracy data for it.

Best for. Large-scale, hard-to-reach data collection where proxies and browser sessions matter. Use it within each target site's terms and the law; this guide does not cover evading access controls. For headless browser options see browser infrastructure for AI agents.

Apify

What it is. A platform and marketplace where developers publish Actors, which are serverless scrapers and automations. An agent can find and run an Actor for almost any site.

Pricing (as of Nov 11, 2026). Free: $0 with $5 of monthly usage, 5 concurrent runs and 16 GB RAM. Starter $19 (CU at $0.20, 32 runs), Scale $199 (CU $0.16, 128 runs) and Business $999 (CU $0.13, 256 runs), each including prepaid credit equal to the monthly fee. Annual billing saves 10 percent. Unused credit expires each cycle; on Free, access stops when credit is gone, on paid plans overage is billed. Individual Actors can add their own fees, so a per-1K price depends on what you run.

Strengths. Coverage: the hosted MCP server at mcp.apify.com loads search-actors, fetch-actor-details and call-actor by default, plus apify/rag-web-browser and apify/web-fetch, so an agent can discover a scraper at runtime. OAuth and bearer-token auth are both supported.

Weaknesses. Quality and pricing vary by Actor author, call-actorreturns run IDs rather than output (you read a dataset afterward), and the default endpoint requires a token. No public search-accuracy benchmark was found.

Best for. Site-specific extraction such as social, maps, e-commerce and job boards, where a maintained Actor beats a generic reader.

Parallel Web

What it is. A web API company building search and research for agents, with Search, Extract, Task, Decisions, Responses and FindAll APIs.

Pricing and limits (as of Nov 11, 2026). Search starts at $1 per 1K requests with 10 results included (extra results $1 per 1K). Extract is $1 per 1K URLs. Task starts at $5 per 1K successful runs, Responses at $10 per 1K. The free Launch tier needs no account; Build gives $5 free monthly with a card on file plus up to $80 at signup (up to 5,000 free requests a month); startups can get up to $250. Search and Extract are limited to 600 per minute on Build and 1,500 on Scale. The pricing page does not map prices to search tiers (Fast, Turbo, Advanced), and the cloro run lists Parallel at $5.00 per 1K on the 100,000-search plan.

Strengths. The anonymous free MCP endpoint at search.parallel.ai/mcp works with no key, and exposes web_search and web_fetch (up to 20 URLs). Parallel leads cloro's SimpleQA table at 97.8 percent and the Vals AI web-search index it cites (Nov 4, 2026, as reported by Parallel).

Weaknesses. Most of its strongest numbers are self-published or from a competitor-run board, and it trails Exa and Perplexity on some independent hard-retrieval tests (77.0 versus 83.0 and 77.7). The standalone search MCP page is marked legacy in favor of a newer combined server that adds research and decisions tools.

Best for. Cheap fast search for agents at volume, and teams that also want task and FindAll style research products.

Linkup

What it is. A search and retrieval API for agents with standard and deep search, a fetch endpoint and asynchronous research.

Pricing (as of Nov 11, 2026), per the docs, in USD. Search standard $0.005 per call ($0.006 for a sourced answer or structured output), deep $0.05 ($0.055). Fetch: standard mode $0.001 without and $0.005 with JS rendering, pro mode $0.005 and $0.01. Research is $0.25, $0.50, $1.50 or $2.50 by size. A structured-schema fetch adds $0.001. New accounts with a professional email get $20, topped up monthly for eligible accounts. The main pricing page advertises 4,000 free queries and up to $5,000 for startups. No rate limits are published.

Strengths. Transparent per-call pricing, an MCP server with search, research, get-research and fetch tools, and installs for Claude Code, Cursor, VS Code, Codex and Claude Desktop. Linkup fast scored 96.7 percent on Openbenchmarks' factual board.

Weaknesses. On the hard-retrieval and multi-hop tasks it sits mid-table (48.3 completion with fetch; F1 about 40 to 42), and deep search at $50 per 1K is the costliest search mode here. Public benchmark coverage is thin.

Best for. Predictable per-call budgeting and simple agent search where you rarely need deep multi-hop research.

MCP Servers

Every API in this guide ships an MCP server, which is the fastest way to give a coding agent live web access. These are the endpoints and tool names from each vendor's docs on Nov 11, 2026. For a wider catalog see best MCP servers for Claude Code and top MCP servers.

If you are wiring several servers into one agent runtime, the sibling guide on MCP integration platforms covers gateways that manage many servers behind one connection. Keep the tool list small: each extra MCP tool is more schema for the model to read on every turn.

Cost per 1K

Normalizing prices is where most comparisons go wrong, because plans, credits and tokens do not line up. This table converts each vendor to a per-1,000 view as of Nov 11, 2026. Derived figures are marked and are our arithmetic from list prices, not quotes.

Two lessons stand out. First, cost per request and cost per solved task diverge: Openbenchmarks reports median cost per hard-retrieval task from $0.050 to $0.153 across vendors, a three-fold spread driven by how many calls the agent needs, not by list price. Second, if your agent reads pages, the extract price dominates. Exa contents at $1 per 1K pages per type, Linkup fetch at $1 per 1K, Parallel extract at $1 per 1K URLs and Perplexity fetch_url at $0.50 per 1K are all cheap compared with the LLM tokens the markdown will consume. See best vector databases for AI agents if you cache pages for later retrieval instead of re-fetching.

Benchmarks

There is no neutral, widely accepted leaderboard for agent search APIs. What exists is a mix of vendor-run tests, a competitor-run open repository and an anonymous-operator aggregator. We list all of them with who ran them and what limits apply. Do not read any single row as a ranking.

How to read this. The cloro table is the most transparent: one search per question, one fixed reader model, one fixed grader, public prompts and graded answers, and 95 percent Wilson intervals. It still comes from a vendor with a product in the test, and its top three overlap statistically. Parallel's tiered BrowseComp is the most agent-like but uses 50 questions, and Parallel chose the frame. Openbenchmarks publishes code and datasets and varies one component at a time, but names no operator and flags its own factual board as biased. On those boards Exa and Perplexity are the most consistent, Parallel and Firecrawl are cost-efficient, and Tavily and Brave are cheaper to start with but lower on hard tasks.

Why rankings flip. SimpleQA rewards a single clean snippet, FreshQA rewards recency, BrowseComp rewards persistence across many searches, and multi-hop tests reward recall. An API with a cheap fast mode wins the budget tiers; an API with strong snippets wins the one-shot tiers. Vendor blog results also tend to appear only for the benchmark and tier where that vendor leads.

How to Choose

If you are building a full agent rather than a single tool call, our sibling guides on AI agent frameworks and AI agent sandboxes cover where search results end up: the orchestration layer that calls these APIs and the isolated runtime where fetched content should be handled.

Setup Example

Add two or three servers, run your twenty hardest real queries through each, and keep the one whose results you would trust. Commands below use the documented endpoints from Nov 11, 2026; replace the key variables with your own.

# Claude Code: add three search/extract MCP servers (hosted, as documented Nov 11, 2026)
claude mcp add --transport http exa https://mcp.exa.ai/mcp
claude mcp add --transport http tavily https://mcp.tavily.com/mcp/
claude mcp add --transport http linkup https://mcp.linkup.so/mcp \
  --header "Authorization: Bearer $LINKUP_API_KEY"

# Brave (local, needs BRAVE_API_KEY)
claude mcp add brave --env BRAVE_API_KEY=$BRAVE_API_KEY -- npx -y @brave/brave-search-mcp-server

# Jina Reader without a key (20 RPM): prepend the URL
curl -s https://r.jina.ai/https://example.com/docs/page

For custom code, write your agent against a tiny search and extract interface so vendors are swappable. This sketch degrades to the snippet when a page cannot be fetched, which keeps one blocked site from failing a whole answer.

// Provider-agnostic search then extract, with a fallback
type Hit = { url: string; title: string; snippet: string };

async function ground(query: string, search: (q: string) => Promise<Hit[]>,
                      extract: (url: string) => Promise<string>) {
  const hits = (await search(query)).slice(0, 5);          // 1 search call
  const pages = await Promise.allSettled(hits.slice(0, 3).map((h) => extract(h.url)));
  return hits.map((h, i) => ({
    ...h,
    markdown: pages[i] && pages[i].status === "fulfilled" ? pages[i].value : h.snippet,
  }));
}

Security note. Fetched pages are untrusted input. A page can carry text written to hijack an agent. Keep tool permissions narrow, never let page text widen them, and require human confirmation before actions that change data or spend money.

How We Evaluated

We fetched each vendor's pricing, docs and MCP pages on Nov 11, 2026, and read the three public benchmark sources directly. Prices are list prices before tax and credits. Where two pages on the same site disagreed (Brave's free credit, Jina's rate limits, Tavily's Project price), we say so instead of picking one. We did not run our own accuracy benchmark, so every score above belongs to the party named beside it. Vendor-claimed numbers are labeled vendor-reported. There is no paid placement, no affiliate link and no sponsored label in this ranking. Vendors that publish no pricing or limit are shown as not published. Findings that we could not confirm from a first-party page, such as legacy Sonar fees and Jina per-token dollars, are left out of the main tables.

FAQ

Sources

All pages accessed Nov 11, 2026.

Pricing and limits

• Firecrawl pricing
• Exa pricing
• Tavily pricing, API credits, rate limits
• Jina Reader and Search
• Brave Search API
• Perplexity pricing, rate limits and tiers, Sonar Pro status
• SerpAPI pricing
• Bright Data SERP pricing
• Apify pricing
• Parallel pricing
• Linkup pricing, pricing page

MCP servers

• Firecrawl MCP
• Exa MCP
• Tavily MCP
• Jina MCP
• Brave Search MCP server
• Perplexity MCP
• SerpAPI MCP
• Bright Data MCP
• Apify MCP
• Parallel Search MCP
• Linkup MCP

Benchmarks and media

• cloro web-search-api-benchmark (vendor-run)
• Parallel: most powerful search API (vendor-reported)
• Openbenchmarks web search
• Firecrawl: most accurate /search yet (vendor-reported)
• Firecrawl intro video
• Exa Connect announcement, Exa MCP update