Your Scraper Isn't Blocked. It's Been Wrong for Three Weeks.

Almost everything written about web scraping reliability is about getting blocked. Rotate your proxies, manage your fingerprints, handle the CAPTCHAs. That's a real problem and we've written about it at length.

It's also the easier problem, because getting blocked tells you it happened. You get a 403, a challenge page, a collapse in throughput. Something fires an alert, someone looks at it, and the clock on the incident starts within minutes.

The more expensive failure is when your scraper runs on schedule, returns 200 OK, writes clean-looking JSON to your warehouse, and has been subtly wrong since a markup change three weeks ago that nobody noticed. Nothing alerts, because from the pipeline's point of view nothing failed.

Two failure modes with very different costs

• Blocked. Detected in minutes. Costs you a gap in the data, which you can backfill once it's fixed.
• Drifted. Detected in weeks or months, usually by accident, when someone looks at a dashboard and says "that can't be right." Costs you every decision made on the bad data in the meantime, plus trust in the pipeline afterward. Often it can't be recovered, because by the time you find it the source pages have moved on.

Teams monitor heavily for the first and almost never for the second. Blocks look like errors, and error monitoring comes largely for free. Drift looks like success.

Why selectors rot

Parsing HTML with CSS selectors or XPath means writing code against a structure the site never promised to keep. It changes for reasons that have nothing to do with you:

• Obfuscated and generated class names. Modern build tooling produces class names like .Nn35F that are outputs of a hashing step, not stable identifiers. They change whenever the site's CSS is rebuilt, which can be any deploy.
• A/B tests. You may be in a bucket, and the bucket may change. This is the nastiest variant, because it produces intermittent wrongness that looks like noise rather than breakage.
• Regional and device layout variants. The same URL can return structurally different markup by country, language, or device, so a parser validated against one configuration can silently miss in another. This is closely related to the targeting problem in our search localization guide: you can be querying a page you never actually tested against.
• Gradual rollouts. A redesign that reaches 5% of traffic corrupts 5% of your rows, which is far below the threshold where anyone notices a trend line move.

None of these produce an exception. A selector that matches nothing returns an empty list, and empty lists serialize to valid JSON.

A 200 status code doesn't mean the data is right

The default health check for a scraping job is "did the request succeed," and it tells you almost nothing about this failure mode. A 200 means the server sent bytes. It says nothing about whether those bytes contained what you think they did.

The same goes for the naive defensive-coding habit that usually accompanies it:

# Looks defensive, but it hides parser failures
try:
    price = soup.select_one(".product-price").text
except AttributeError:
    price = None   # swallowed and now indistinguishable
                   # from a product that genuinely has no price

Every broad except around a parse step converts a loud failure into a silent one. A month later, price IS NULL on 40% of rows, and nobody can tell whether that's a data property or a parser that stopped working, because the information needed to tell them apart was discarded at extraction time.

Design your pipeline to detect drift

Better selectors won't fix this, because selectors will rot regardless. The goal is to find out quickly when they do.

Canary queries. Keep a small set of inputs whose correct output you know and that shouldn't change day to day. Run them on every cycle and assert on the values, not just the shape. When a canary that returned the same answer for six months suddenly doesn't, you have a dated signal pointing at the change, which is exactly what you lack when drift is discovered by accident.

Assert the contract, don't swallow the miss. Decide which fields are genuinely optional and which are structural, then fail loudly on the structural ones. A missing price on a product page is a parsing failure worth an alert; a missing "sale ends" badge probably isn't. The point is to make that a deliberate, documented choice rather than the accidental consequence of a broad except. This matters even more if you skip re-processing pages that haven't changed: a parser that matches nothing produces the same empty fingerprint on every visit, so the page looks "unchanged" forever. Our incremental scraping guide covers how to guard against that.

Monitor distributions as well as errors. This is the most valuable monitoring you can add for the effort, and most pipelines have none of it:

• Fill rate per field: what fraction of rows have a non-null value. A drop from 98% to 0% is a broken selector; a drop to 60% is a rollout in progress.
• Result counts per request: if a query that reliably returned ten results starts returning three, something changed upstream.
• Value shape: string lengths, numeric ranges, format validity. Prices that parse as numbers but are suddenly 100× larger mean you're reading a different element, or a different currency.

Alert on movement in these numbers rather than on fixed thresholds. The question you're always asking is whether the page changed underneath you.

Keep the raw response. When you do detect drift, the first question is when it started, and you can only answer that if you kept the raw HTML alongside the parsed output. Storage is cheap relative to a quarter of unusable data. This is also what makes recovery possible at all: with raw responses retained, fixing the parser lets you re-run it over history rather than accepting a permanent hole.

Where a parsed API moves the burden

Using an API instead of raw HTML doesn't remove all of this work. You still own validation of whether the data makes sense for your use case. It does take over the largest piece, because the parsing layer is the part that rots.

PrismCrawl provides documented search and data APIs across search engines, shopping, maps, travel, reviews, and app stores. Enterprise and production customers consume those response contracts instead of maintaining their own page selectors. For Google and Bing search specifically:

• Required fields are required. results, serp_features, search_parameters, and has_next_page are structural guarantees in the schema, not fields that might quietly stop appearing. A change in what Google renders is our problem to absorb, not a silent null in your warehouse.
• confidence on extracted features. SERP feature extraction carries a confidence score, so ambiguous parses arrive labelled as ambiguous rather than presented as certain.
• search_parameters echo back. Every parameter you sent returns with the response, so a stored row remains interpretable later; you can tell what was actually asked, which is half of any drift investigation.
• Retained artifacts for replay. Request metadata is kept for 90 days and paired HTML and JSON artifacts for 30, so when you find a discrepancy you have the original response to check against instead of a parsed row and a guess. (Note that zero_trace deliberately skips archiving. That's useful for sensitive queries, but it opts you out of this.)
• Errors are free. Failed searches don't consume credits, which removes the small but real incentive to swallow a failure rather than surface it.

If you're weighing this against maintaining your own parsers, the build vs. buy breakdown covers the cost side. Parser maintenance is the line item teams underestimate most, because its failures don't show up in an error budget. The free tier is 100 credits with no credit card if you want to test the response shape against your own validation logic first.

Frequently asked questions

How do I know if my scraper is silently returning bad data?

You won't find out from error rates, because there aren't any. The practical approach is to monitor the shape of the data rather than the success of the request: track field fill rates, result counts, and value distributions over time, and alert when they shift. A parser that quietly stops matching an element shows up as a fill rate dropping from 98 percent to zero, not as an exception.

Why isn't a 200 status code enough to confirm a scrape worked?

A 200 only tells you the server sent a response. It says nothing about whether your parser understood it. A site can redesign its markup, serve you an A/B variant, or return a different layout for a different region, and every one of those cases produces a perfectly valid 200 response that your selectors no longer match. Status codes measure delivery, not correctness.

Does using a scraping API eliminate this problem?

It moves the burden rather than eliminating it. A parsed API means you're consuming a versioned response contract instead of maintaining selectors against markup that changes without notice, so the parsing layer becomes somebody else's operational problem. You still own validation of whether the data makes sense for your use case, and you should still monitor for distribution shifts in what you receive.