Collecting External Web Data to Drive Automations: Where Scraping Fits

Plenty of automations stall at the same point: the fact needed to decide lives on somebody else's website. The question is not how to scrape it but where to take it from, because every source has a different reliability - and an automation is worth exactly as much as the data that fires it.

Four sources, four levels of reliability

That order is not an aesthetic preference. An official interface commits to a structure, announces changes, and returns a clear error when something is wrong; a web page commits to nothing. The full comparison between the two ends is in web scraping against an API.

What breaks when external data drives an automation

The problem is not the collection but what follows it. Three failures recur:

• Triggering on stale data - the collection failed, the old value remained, and the automation acts on it as if it were current.
• Flooding - every tiny change fires an action, and after two days nobody reads the alerts.
• Duplicate action - the same record is collected again and produces the same action again.

The first is the serious one, and also the easiest to prevent: store the collection time alongside every value, and do not act on a value past a threshold age. "No data" and "week-old data" should be two different states, and both different from "all is well".

The rule that prevents most incidents: an unknown state

An external data pipeline needs three states rather than two: valid value, stale value, and unknown. Most systems implement two - there is a value or there is not - and then the absence of a value is wrongly read as a business fact.

The concrete example: tracking stock at a supplier. If the scraper fails and the system sees "no data", it may mark the product as out of stock and halt sales. With an explicit third state - "we could not check" - the outcome is an alert to a person, not an automatic action.

What to store alongside every value

The value itself is not enough. Four extra fields turn a fragile pipeline into one you can rely on:

The last field is what makes the third state described above possible. Without it, a partial run that collected 200 of 4,000 records looks exactly like a successful run against a source that emptied out.

How to define a trigger that does not flood

• Set a materiality threshold: a change below a given percentage is not an event.
• Batch changes into one digest rather than an alert each.
• Alert on the absence of change when that is unusual - a frozen source is a sign of a fault.
• Separate "worth knowing" from "needs action", and send them on different channels.
• Decide who receives each type, and confirm they actually read it.

The third item catches silent failures. A source that has not changed in two weeks is not necessarily stable - it is often a scraper returning the same cache. An alert on stagnation is the cheap way to find that out. For the basics, see what web scraping is.

Idempotency: why this is not a luxury

Every such pipeline will process the same record twice - because of a retry, a stalled run, or a change that looks like a new record. If the action at the end is sending an email or creating a task, that means a duplicate in front of a customer. This is what I do for clients - see web scraping services.

The fix is simple and not obvious: derive a stable identifier from each record and record it before performing the action. An action whose identifier already exists is skipped. That costs one database field and prevents the class of fault discovered by customers rather than by tests.

Three examples that show the difference

Tracking a competitor's price. The value changes slowly and the action - a price update - is expensive to reverse. So: a daily frequency with a materiality threshold, and no automatic action on a value more than a day old.

Stock at a supplier. The value changes fast and the action - halting sales - is dramatic. Here the third state matters most: when we could not check, alert rather than halt.

A new item in a feed. The value is an event rather than a figure, so there is no "stale" - only "already handled". Here the key is a unique identifier per item, not a comparison against a previous value.

All three use the same pipeline with entirely different rules, which is exactly why these decisions belong to the design stage and not to the code.

Where this sits in the wider picture

Collection is one stage of four: collect, normalise, store, act. The instinct is to invest everything in the first, while most real incidents sit in the second and third - a changed format, different units, a record counted twice.

That is why even a small pipeline is worth building as separate stages rather than one script: when the source changes, you fix one stage rather than everything. The full breakdown is in a data pipeline for scraped data.

Sources

• robots.txt - the Robots Exclusion Protocol
• Web scraping against an API
• A data pipeline for scraped data
• Book a call