Why Web Scrapers Break, and What Maintenance Actually Covers

A scraper does not break at random - it breaks in seven familiar ways, and nearly all of them come from a change on a site nobody announced. What separates a reliable system from an unreliable one is not better code but that it shouts when the data stops being correct.

The seven ways a scraper breaks

The sixth row is the dangerous one. When a site changes its date format or starts quoting prices including tax, the scraper keeps running successfully and feeds wrong data into a system that trusts it - and nobody knows until somebody notices an odd decision.

Why a silent failure is worse than a crash

A scraper that crashes produces an error, and the error produces a response. A scraper returning 40 records instead of 4,000 keeps updating the database, and the systems consuming it keep working - on a partial picture.

That explains why the right investment is not in more resilient code but in monitoring. Code cannot know that a site changed meaning; monitoring can know that something looks different from yesterday.

The three measures that catch nearly everything

• A count floor - a run returning less than X percent of the previous run counts as a failure, even with no error raised.
• Empty-field rate - a sudden jump on one field points precisely at the selector that broke.
• A canary record - one known record collected on every run, with its values compared against expected ones.

The third is the cheapest and strongest. Pick one stable product, record its correct values, and compare on every run. When its price is suddenly "0" or its title is empty, you know within minutes that something changed - without inspecting 4,000 records.

What maintenance actually covers

This is where expectations collide. Many clients read "maintenance" as bug fixing, and in practice most of the work is not in your code but in adapting to what changed on the other side:

• Updating selectors after a page structure change.
• Adapting to a change in pagination or navigation path.
• Handling new defences deployed at the source.
• Adapting to a change in meaning - currency, units, format.
• Cleaning and standardising when the source starts returning new values.
• Periodically confirming the data still matches what the browser shows.

The last item is the one people skip and the only one that catches a silent "meaning" failure. Manually comparing five records against the site, once a month, is the highest-return check on the whole list.

Why a retainer is the sector default

A data source is not a contract - the site does not undertake to keep its structure, and does not give notice. So a scraper is a maintained system, exactly like an integration against a supplier who updates their API.

The practical difference between working with a retainer and without one is not cost but response time. With no arrangement in place, a site change produces an enquiry, a scheduling call and a quote - and meanwhile the data is stopped. Background on how the cost is built is in how much web scraping costs.

What reduces breakage up front

• Depend on stable identifiers rather than on position in the page.
• Prefer structured data in the page where it exists, before parsing text.
• Keep a raw copy of the page for each run, so what happened can be reconstructed.
• Split into stages - collect, parse, load - so a failure in one does not contaminate the rest.
• Cross-check against an official source where one exists, even while scraping.

The third item turns investigating a fault from guesswork into fact. With stored HTML from the failed run you can see exactly what arrived; without it you are reconstructing against a site that has already changed again. The broader pipeline side is covered in a data pipeline for scraped data.

How often does this actually happen?

There is no single number, and it depends almost entirely on the source: a small business site can stay stable for two years, while a large platform running design experiments changes something every few weeks. What can be said is that the rate at a given source is fairly steady, so after six months you know what to expect.

What does not change is that the change is never announced. So the right question is not "will it break" but "how long will it take us to know".

Sources

• robots.txt - the Robots Exclusion Protocol
• A data pipeline for scraped data
• How much web scraping costs
• Book a call