PrismCrawl
← Back to blog

Published

Your AI Agent Doesn’t Need 60 Scraping Tools. It Needs Three Good Ones.

There is a familiar way to make an AI agent look powerful in a demo: connect every API, scraper, browser action, and site-specific extractor you can find. The model opens its toolbox and sees dozens of ways to search Google, download a page, render JavaScript, take a screenshot, crawl a site, extract a product, or click a button.

That looks like capability. To the model, it often looks like a multiple-choice test with 60 almost-correct answers.

Every tool has a cost before it runs. Its name, description, parameters, enums, and examples have to be made available to the model somehow. Overlapping tools create a routing problem. Large responses then consume more context after the call. A task that should have been “find three sources and compare them” becomes an expensive loop in which the agent first has to understand the scraping platform itself.

For most web-research agents, a better starting point is only three capabilities:

  1. Search to discover candidate sources.
  2. Fetch to read a selected source.
  3. Extract to return known fields as structured data.

The hard part is not naming those three tools. It is deciding what each tool is allowed to return, when the agent should move to the next one, and what stays outside the context window.

Why MCP web-scraping servers can expose too many tools

Suppose an agent can call all of these:

google_search          search_engine         search_news
scrape_url             fetch_page            scrape_as_markdown
render_page            browser_navigate       extract_page
extract_product        scrape_product         get_product_data

A developer can probably explain the difference between them. The model has to infer it from the definitions available during that turn. Does fetch_page render JavaScript? Does scrape_url return HTML or text? Is get_product_data cheaper or more accurate than extract_page with a schema? Which search operation supports location?

None of those questions advances the user's task. They are interface ambiguity imposed by the tool provider.

This is not just theoretical. Bright Data says the first version of its web MCP exposed more than 60 tools, but roughly 90% of agent calls used only search and scrape-as-Markdown. It also reports that listing the full toolset cost about 17,000 tokens before useful work began. Its default “Rapid mode” now exposes only those two common operations, with the larger surface available as an opt-in. Bright Data's account of that redesign is valuable because it describes behavior observed after shipping, not merely an aesthetic preference for small APIs.

Massive reached a similar conclusion in a different data domain. Its MCP server once mapped 53 API endpoints to 53 tools. The company replaced them with four composable operations: search endpoints, read documentation, call the API, and query stored data. It reports cutting context overhead by more than 90%. The agent discovers the detailed API surface only when it needs it. Massive's MCP redesign is progressive disclosure applied to tools.

The lesson is not that three is a magical number. It is that capabilities should be broad and distinct at the top level, while complexity is revealed on demand.

The three-tool web architecture

A useful contract might look like this:

type SearchWeb = (input: {
  query: string;
  country?: string;
  language?: string;
  limit?: number;
}) => Promise<{
  results: Array<{
    title: string;
    url: string;
    snippet: string;
    rank: number;
  }>;
}>;

type FetchPage = (input: {
  url: string;
  format?: "text" | "markdown";
  maxCharacters?: number;
}) => Promise<{
  url: string;
  title?: string;
  content: string;
  fetchedAt: string;
  truncated: boolean;
}>;

type ExtractData = (input: {
  url: string;
  schema: Record<string, unknown>;
}) => Promise<{
  url: string;
  fetchedAt: string;
  data: unknown;
  validationErrors: string[];
}>;

These tools do different jobs. Their boundaries are easy for a model to recognize, and their outputs match different stages of research.

1. Search discovers; it does not prove

Search is the cheapest way to reduce the open web to a shortlist. It returns titles, snippets, URLs, ranks, and perhaps dates or result types. That is enough to locate primary documentation, recent reporting, competing claims, or pages likely to contain the requested data.

It is not enough to claim that the underlying page says something its snippet only hints at. Search results are discovery data. When a conclusion depends on a page's details, the agent should select the best result and fetch it.

PrismCrawl fits at this first layer. It gives an agent live Google or Bing results as structured JSON without making the agent manage search-page browsers, proxies, CAPTCHAs, or result selectors. The agent receives a small ranked set instead of an entire search page's HTML. Our live-search integration guide shows the wrapper; the architectural point here is that search should remain a focused, predictable capability.

2. Fetch reads only what earned a place in context

Fetch is for the few pages that survive the search step. Its default output should be readable content, not raw HTML, and it should enforce a size limit. It should also say when a response was truncated.

The cheapest page is the page the agent never fetches. If ten search results contain two promising primary sources, fetching all ten “just in case” adds latency, scraping cost, and thousands of irrelevant tokens. Fetch two, assess what is missing, and expand only if the evidence is insufficient.

Even clean Markdown is not automatically small. Navigation, repeated footers, related articles, and long tables can still crowd out the passage that matters. Useful fetch implementations support section selection, character limits, or a follow-up operation that reads a specific range from content stored outside the prompt.

3. Extract returns records, not reading material

Extraction is appropriate when the desired fields are known: product name and price, event date and venue, job title and location, or an article's author and publication date. The caller supplies a schema; the tool returns data that either validates against it or includes explicit validation errors.

Do not make the model read a 100,000-token catalogue to recover six fields from every row. Extract near the source, validate there, and send only the useful records onward.

ScrapingBee tested several representations of the same catalogue page. In its published experiment, raw HTML used 9,673 tokens while selector-based extraction returned 623 at the same scraping-credit cost. Its broader conclusion is neatly phrased as “carry control, not data”: large datasets belong in storage, while the model receives a handle and queries what it needs. The full ScrapingBee benchmark includes the methodology and its version-specific measurements.

Search, fetch, and extract are a ladder

The important design is the escalation policy between those three boxes:

Question
   ↓
SEARCH: Which sources are likely to answer it?
   ↓ select a small number
FETCH: What do those sources actually say?
   ↓ only when repeated fields are required
EXTRACT: Which schema-validated records should enter the workflow?
   ↓
Answer with source URLs and retrieval times

Each step earns the next one. A browser can sit above this ladder as an exceptional capability for interaction, authentication, infinite scroll, or content that a normal fetch cannot retrieve. It does not need to be the default path for every question.

Site-specific extractors can work the same way. Keep them behind a capability search or a router. If the agent asks for 5,000 product records, the system can reveal a purpose-built commerce extractor without loading every vertical tool into every unrelated research session.

Oxylabs takes a related approach with agent skills intended to teach agents how to use its scraping interfaces without inventing parameters or misusing the API. That addresses a different layer of the same problem: an agent needs operational knowledge as well as connectivity. Oxylabs describes the goal as reducing API hallucinations. A small top-level interface makes those instructions easier to scope and load only when relevant.

Keep the data plane out of the conversation

An MCP server or function call is a control interface. It does not follow that every byte produced by the underlying system belongs in the conversation.

For a small article, returning clean text may be reasonable. For a crawl containing thousands of pages, the tool should instead return something like:

{
  "dataset_id": "crawl_01K4...",
  "rows": 12840,
  "schema": ["url", "title", "price", "currency", "in_stock"],
  "sample": [
    {
      "url": "https://example.com/p/1",
      "title": "Example product",
      "price": 29.95,
      "currency": "USD",
      "in_stock": true
    }
  ],
  "next": "Use query_dataset with dataset_id to filter or aggregate these rows."
}

The model sees the shape, size, sample, and handle. The records stay in object storage, a database, or a sandbox where code can filter and aggregate them. If the user asks for the median price, compute it beside the dataset and return the number plus the query, not 12,840 rows for the model to count.

This separation also improves recovery. Raw responses can be retained outside context for auditing or re-parsing, which matters when a site's markup changes. As discussed in our guide to silent scraper failures, storing raw input can make historical repair possible. Retaining it is different from feeding it to the model on every turn.

Small outputs are also a security boundary

Web pages are untrusted input. A page can contain instructions addressed to a model, text hidden from human readers, misleading metadata, or simply thousands of irrelevant words. Passing the whole page into the agent gives all of it an opportunity to influence later reasoning.

A smaller interface cannot eliminate prompt injection, but it reduces exposure:

  • Search returns a fixed set of factual fields.
  • Fetch labels page content as untrusted evidence and limits its size.
  • Extract validates output against a schema and rejects unexpected fields.
  • Credentials, cookies, internal prompts, and tool instructions never appear in results.
  • The application, not the page, decides which actions are permitted next.

The key rule is that retrieved content supplies data, not authority. A sentence on a page cannot expand a tool's permissions, change the research objective, or authorize another external action.

Write contracts that make failure visible

Minimal tools should not mean vague tools. Each result needs enough operational metadata for the agent and application to tell success from plausible-looking failure.

A good result contract includes:

  • the final source URL and retrieval time;
  • whether JavaScript rendering was used;
  • whether content was truncated;
  • a stable request or trace ID;
  • structured error categories such as blocked, timeout, invalid schema, or empty content;
  • validation failures instead of invented placeholder values;
  • usage data when the agent has a request or cost budget.

This is especially important for extraction. 200 OK means a response arrived; it does not mean the expected product grid, article body, or search results were present. Require content assertions such as a minimum result count, required fields, or plausible value ranges.

Measure the agent, not the size of its toolbox

Tool count is an easy launch metric and a poor production metric. Measure whether the interface helps the agent finish grounded tasks efficiently:

  • Tool-selection accuracy: Did it choose search, fetch, or extract at the right stage?
  • Unnecessary-call rate: How often did it fetch pages that contributed no claim?
  • Tokens per supported claim: How much context produced evidence that survived into the answer?
  • Citation coverage: What percentage of externally derived claims point to a source?
  • Duplicate-source rate: Did repeated queries return the same evidence under different URLs?
  • Schema-valid extraction rate: How many records passed validation without repair?
  • Budget adherence: Did the run remain inside its search, page, token, and time limits?

Build a small evaluation set from real tasks. Record the expected route as well as the expected answer: search only, search then fetch, or direct structured extraction. Run it whenever a tool name, description, schema, or routing rule changes. An interface rewrite can change agent behavior even when every underlying scraper still works.

Start with three, then make complexity earn its place

Search, fetch, and extract cover a large share of agentic web research because they correspond to three genuinely different intentions: find, read, and structure. They are easy to explain, easy to budget, and easy to observe.

Add a browser when interaction is required. Add vertical extractors when their accuracy or economics justify them. Add dataset-query tools when results outgrow the context window. But reveal those capabilities when the task calls for them instead of making every agent understand the entire scraping stack before its first search.

The best web-data layer is not the one with the most impressive tool list. It is the one that gives the model the smallest useful next decision and returns exactly enough evidence to make that decision well.

Use PrismCrawl as the search layer, review the API reference for its structured Google and Bing response formats, or read how to connect it to an AI agent.

Frequently asked questions

What scraping tools does an AI agent actually need?

Most research agents can start with three capabilities: search to discover sources, fetch to read selected pages, and extract to turn known fields into structured records. Specialized browser or vertical tools can remain hidden until a task requires them.

Why can too many tools make an AI agent less reliable?

Tool names, descriptions, and schemas consume context and create competing choices. When many tools overlap, the model must first decide among nearly identical operations, increasing token use and the chance of an unnecessary or incorrect call.

Should a scraping tool return raw HTML to an AI agent?

Usually not. Raw HTML contains markup, navigation, scripts, and other content irrelevant to the task. Return compact text for reading or schema-validated JSON for known fields, while storing the raw response outside the model context when it is needed for auditing or reprocessing.

Related posts