Competitor Monitoring: Treat a Website, Product, or Message Change as an Event Requiring Context

"The page changed" is not information. A useful event says what changed, in which layer - site, product or message - what the value was before, when you observed it, and which decision it touches. Without those five fields you get an alert stream that is read for two weeks and then filtered to a folder.

A change is not news until it has context

A system that announces "the competitor's pricing page changed" hands all the work back to the reader: open it, compare by eye, guess what moved. After three of those, nobody opens the fourth.

An actionable event looks different. It says: on the pricing page, in the mid-tier plan row, the number moved from 249 to 299, observed on 14 August at 07:12, and this touches your own pricing. The same quantity of technology, a completely different level of usefulness.

The difference is created during scoping, not during development: you decide in advance which regions of a page count as meaningful, and what the business name of each region is. A region with no business name will only ever produce a generic alert.

What does a change event record actually contain?

Eight fields turn a notification into something someone can act on:

• source_url - the exact address that was read, not the name of the site.
• region - a business name for an area of the page: "price row - mid tier", "customer logo list", "main headline".
• previous_value and current_value - the normalised values, as short text.
• observed_at - your observation time in UTC, not the publication time the site displays.
• previous_observed_at - when the earlier value was seen. The gap between the two is your uncertainty resolution.
• layer - site, product or message.
• evidence_ref - a pointer to the stored raw capture, for when someone claims the system got it wrong.
• confidence - whether the region was located by a stable anchor or by a positional guess.

The field whose absence hurts most is previous_observed_at. Without it you cannot say "the price went up yesterday" - you can only say "the price differs from the last time we looked", and those are two entirely different claims.

Normalising before the diff: what you delete so it stops shouting

Diffing raw HTML produces an alert on nearly every run. The noise comes from predictable places, and deleting them before the comparison is the whole job:

• Timestamps and dates rendered into the page, including "updated 3 hours ago".
• Session ids, CSRF tokens and request ids embedded in links and tags.
• Build-hashed filenames - a stylesheet that gets a new name on every deploy.
• Regions that rotate content: recommendations, "recently viewed", banners, carousels.
• Stock counters or view counters that move every minute.
• Whitespace, attribute order and quoting differences produced by rendering rather than by an edit.

After normalising, compare text rather than markup. The cheap way to store that is a hash per region, and compare hashes. The Internet Archive does exactly this in its CDX interface: each row returns urlkey, timestamp, original, mimetype, statuscode, digest and length, where digest is a hash of the captured payload. Two rows with the same digest are the same content, and that is the cheapest possible check.

Three change layers, three reactions

That table is a filtering tool, not a catalogue. An organisation that monitors all five rows without deciding who receives each row has built five noise channels. The row most organisations can drop is "site": a new page is usually routine content work, not a move.

Which signal finds a change first - the sitemap, a feed, or the page?

In order, cheapest to most expensive:

1. sitemap.xml. The Sitemaps protocol defines <lastmod> as the date of last modification of the file, and allows an index file pointing at sub-files. One download reveals added pages, removed pages and declared modification dates. Remember that lastmod is asserted by the site itself and is therefore not necessarily accurate.
2. An RSS or Atom feed. Where one exists, it hands you items with timestamps instead of guesswork.
3. A conditional request for the page. The HTTP standard defines ETag and Last-Modified on the response, and If-None-Match and If-Modified-Since on the request. A 304 response means "unchanged" without transferring the body - which saves bandwidth for you and for a server that is not yours.
4. A full fetch and diff. Only when none of the above exists.

That ordering is not a performance trick. It is also a polite rate: a site that can answer 304 saves itself work. The obligation to check each source's directives and terms stays with you, and it is set out in the legal and technical checklist for competitor monitoring.

The three failure modes of change monitoring

• Alerting on nothing. Almost always caused by skipping normalisation. The symptom is an alert on every run, in the same region, with no visible difference.
• Silence after the reader breaks. The site moved to different rendering, the selector no longer matches, and the code compares an empty string to an empty string - reporting "no change" forever. The defence is a liveness check: if a defined region returned empty, that is a fault, not a result. The same idea is worked through in silent failure in Zapier and Make automations.
• A correct event to the wrong person. A terms-of-use change routed to a sales rep disappears. Routing belongs in the scoping document, not in a settings screen filled in later.

From alerts to a briefing: what you do with the events

A single alert is worth less than a weekly summary that shows a sequence. Three price changes in a month is a pattern; one is noise. So you keep a series, not just the current state.

A structure that works: a weekly digest grouped by layer, where each event carries its previous and current value plus a link to the stored capture, with one line of human interpretation above it. The interpretation is a person's job, and a model can compress the material before them - the limits of that are described in researching competitors with AI. The broader survey of what can be watched on a competitor's site at all is in monitoring competitor websites for changes.

Sources

• Sitemaps XML protocol - lastmod and sitemap index files
• RFC 9110 - HTTP Semantics, conditional requests and ETag
• Wayback CDX Server API - response fields including digest