Your Scraper Isn't Blocked. It's Been Wrong for Three Weeks.

Almost everything written about web scraping reliability is about getting blocked. Rotate your proxies, manage your fingerprints, handle the CAPTCHAs. That's a real problem and we've written about it at length.

It's also, in a sense, the easy problem — because getting blocked tells you it happened. You get a 403, a challenge page, a collapse in throughput. Something fires an alert, someone looks at it, and the clock on the incident starts within minutes.

The expensive failure is the other one. Your scraper runs on schedule, returns 200 OK, writes clean-looking JSON to your warehouse, and has been subtly wrong since a markup change three weeks ago that nobody noticed. Nothing alerts, because from the pipeline's point of view nothing failed.

Two failure modes, wildly different costs

The asymmetry is worth stating plainly, because it's the whole argument:

Blocked. Detected in minutes. Costs you a gap in the data, which you can backfill once it's fixed. Loud, bounded, recoverable.
Drifted. Detected in weeks or months, usually by accident — someone eyeballs a dashboard and says "that can't be right." Costs you every decision made on the bad data in the interim, plus the credibility of the pipeline afterward. Quiet, unbounded, and frequently not recoverable, because by the time you find it the source pages have moved on.

Teams instrument heavily for the first and almost never for the second. The reason is structural: blocks look like errors, and error monitoring is a solved thing you get largely for free. Drift looks like success.

Why selectors rot

Parsing HTML with CSS selectors or XPath means writing code against a structure the site never promised to keep. It changes for reasons that have nothing to do with you:

Obfuscated and generated class names. Modern build tooling produces class names like .Nn35F that are outputs of a hashing step, not stable identifiers. They change when the site's CSS is rebuilt — which can be any deploy.
A/B tests. You may be in a bucket, and the bucket may change. This is the nastiest variant, because it produces intermittent wrongness that looks like noise rather than breakage.
Regional and device layout variants. The same URL can return structurally different markup by country, language, or device — so a parser validated against one configuration silently misses in another. This is closely related to the targeting problem in our search localization guide: you can be querying a page you never actually tested against.
Gradual rollouts. A redesign that reaches 5% of traffic corrupts 5% of your rows, which is far below the threshold where anyone notices a trend line move.

None of these produce an exception. A selector that matches nothing returns an empty list, and empty lists serialize to valid JSON.

HTTP status measures delivery, not correctness

The default health check for a scraping job is "did the request succeed," and it's close to worthless for this failure mode. A 200 means the server sent bytes. It says nothing about whether those bytes contained what you think they did.

The same goes for the naive defensive-coding habit that usually accompanies it:

# This looks careful. It is the bug.
try:
    price = soup.select_one(".product-price").text
except AttributeError:
    price = None   # swallowed — and now indistinguishable
                   # from a product that genuinely has no price

Every broad except around a parse step converts a loud failure into a silent one. A month later, price IS NULL on 40% of rows, and nobody can tell whether that's a data property or a parser that stopped working — the information needed to distinguish them was discarded at extraction time.

Design for detection, not just extraction

The fix isn't better selectors. Selectors will rot regardless; the goal is to find out quickly when they do.

Canary queries. Keep a small set of inputs whose correct output you know and that shouldn't change day to day. Run them on every cycle and assert on the values, not just the shape. When a canary that returned the same answer for six months suddenly doesn't, you have a dated signal pointing at the change — which is exactly the thing you lack when drift is discovered by accident.

Assert the contract, don't swallow the miss. Decide which fields are genuinely optional and which are structural, then fail loudly on the structural ones. A missing price on a product page is a parsing failure worth an alert; a missing "sale ends" badge probably isn't. The point is to make that a deliberate, documented choice rather than the accidental consequence of a broad except.

Monitor distributions, not just errors. This is the highest-value instrumentation for the money, and most pipelines have none of it:

Fill rate per field — what fraction of rows have a non-null value. A drop from 98% to 0% is a broken selector; a drop to 60% is a rollout in progress.
Result counts per request — if a query that reliably returned ten results starts returning three, something changed upstream.
Value shape — string lengths, numeric ranges, format validity. Prices that parse as numbers but are suddenly 100× larger mean you're reading a different element, or a different currency.

Alert on movement in these, not on thresholds you set once. The signal you want is change, because the underlying question is always "did the page change out from under us."

Keep the raw response. When you do detect drift, the first question is when it started — and you can only answer that if you kept the raw HTML alongside the parsed output. Storage is cheap relative to a quarter of unusable data. This is also what makes recovery possible at all: with raw responses retained, fixing the parser lets you re-run it over history rather than accepting a permanent hole.

Where a parsed API moves the burden

Nothing above stops being your job just because you're consuming an API instead of raw HTML — you still own validation of whether the data makes sense for your use case. But it does move the largest piece of it, because the parsing layer is the part that rots.

PrismCrawl returns Google and Bing search results against a documented response contract rather than markup you have to reverse-engineer. Concretely, for this problem:

Required fields are required. results, serp_features, search_parameters, and has_next_page are structural guarantees in the schema, not fields that might quietly stop appearing. A change in what Google renders is our problem to absorb, not a silent null in your warehouse.
confidence on extracted features. SERP feature extraction carries a confidence score, so ambiguous parses arrive labelled as ambiguous rather than presented as certain.
search_parameters echo back. Every parameter you sent returns with the response, so a stored row remains interpretable later — you can tell what was actually asked, which is half of any drift investigation.
Retained artifacts for replay. Request metadata is kept for 90 days and paired HTML and JSON artifacts for 30, so when you find a discrepancy you have the original response to check against instead of a parsed row and a guess. (Note that zero_trace deliberately skips archiving — useful for sensitive queries, but it opts you out of exactly this.)
Errors are free. Failed searches don't consume credits, which removes the small but real incentive to swallow a failure rather than surface it.

If you're weighing this against maintaining your own parsers, the build vs. buy breakdown covers the cost side — and parser maintenance is the line item that gets underestimated most, precisely because its failures don't show up in an error budget. The free tier is 25 credits with no credit card if you want to test the response shape against your own validation logic first.

Frequently asked questions

How do I know if my scraper is silently returning bad data?

You won't find out from error rates, because there aren't any. The practical approach is to monitor the shape of the data rather than the success of the request: track field fill rates, result counts, and value distributions over time, and alert when they shift. A parser that quietly stops matching an element shows up as a fill rate dropping from 98 percent to zero, not as an exception.

Why isn't a 200 status code enough to confirm a scrape worked?

A 200 only tells you the server sent a response. It says nothing about whether your parser understood it. A site can redesign its markup, serve you an A/B variant, or return a different layout for a different region, and every one of those cases produces a perfectly valid 200 response that your selectors no longer match. Status codes measure delivery, not correctness.

Does using a scraping API eliminate this problem?

It moves the burden rather than eliminating it. A parsed API means you're consuming a versioned response contract instead of maintaining selectors against markup that changes without notice, so the parsing layer becomes somebody else's operational problem. You still own validation of whether the data makes sense for your use case, and you should still monitor for distribution shifts in what you receive.