A comparison results page is a timestamped observation, not a price list. Which fields each capture carries, and how to avoid republishing a catalogue.
Key takeaways
- The base record is not "the price of a product" but an observation: price, currency, seller, URL, capture method and timestamp.
- A price with no `observed_at` is a figure nobody can falsify. The moment someone asks "as of when", the answer has to live in a field, not in memory.
- Identifying a product by its name produces wrong matches. A stable identifier - a GTIN barcode or a manufacturer part number - is what makes a comparison mean anything.
- Storing a raw capture internally as evidence is one thing. Republishing somebody else's catalogue is an entirely different thing.
- Where an official feed exists - food-retail price transparency files, the supplier's own product feed, a partner interface - that is the route, not reading a results page.
A price-comparison results page is an observation: one price, at one seller, at one moment, as rendered to one browser. It is not a price list and it is not an offer. Any design that treats it as a price list eventually shows an old number as though it were true today.
Why a results page is not a price list
A comparison engine renders a list assembled for one request: particular filters, a particular sort, a particular location and sometimes a particular user. Two people opening the same page in the same minute can see a different order, and even different sellers.
So the claim "the price of this product is 899" cannot be verified, while "on 14 August at 09:03, a price of 899 was displayed by seller X on the page at URL Y" is a claim that can be checked. The difference is not semantic: it decides whether a report built from the data survives a meeting.
Three more reasons a results page is not a price list: a displayed price usually excludes shipping, a promotion or a quantity condition; a seller can change the price between the moment you observed it and the moment the report is read; and a row on the page may be a paid placement rather than an organic result. The wider commercial consequences of that gap are covered in monitoring competitor prices in Israel.
What one observation record must carry
| Field | Example | Why the figure is meaningless without it |
|---|---|---|
observed_price | 899.00 | The number as displayed, unrounded and unconverted |
currency | ILS | Comparing across currencies without it is a silent error |
price_includes_vat | true | A consumer price and a business price are not the same number |
seller_name | The seller as shown | An observation with no seller cannot be verified |
listing_url | The specific listing address | Lets a person open it and check |
observed_at | UTC timestamp | Without it no conclusion has standing |
capture_method | "results page" / "supplier feed" / "manual entry" | Prices from different routes carry different certainty |
product_key | GTIN barcode or manufacturer part number | Without it the comparison may be between two different items |
shipping_note | "shipping 29" or empty | A cheap item with expensive delivery is not cheap |
raw_ref | Identifier of the stored capture | Evidence for when someone disputes the number |
The field that surprises people is capture_method. A price that arrived from an official supplier feed and a price read off a display page are not equivalent, and once they sit in one table without that distinction, a contradiction between them can no longer be explained.
This field shape is not an invention. The schema.org vocabulary for an offer draws exactly these distinctions: price, priceCurrency, availability, seller, url and priceValidUntil - meaning even the public vocabulary accepts that a price has an expiry.
When does a stored price become too old to display?
There is no single number, but there is a rule: validity is set by how fast the source moves, not by what is convenient to display. Three practical bands:
- Electronics and consumer goods on comparison engines - move on a scale of days, and hours during promotional periods. A week-old observation is history, not a price.
- Supermarket food items - published in official files on a daily cycle, so there is an external anchor for the frequency.
- Professional equipment and supplier price lists - more stable, sometimes for months, which is exactly where people forget the price moved at all.
The display rule follows: never show a stored price without the time it was observed, and once an observation passes a threshold you defined in advance, render the number greyed with "observed 9 days ago" or do not render it at all. A screen showing "899" with no date quietly manufactures wrong decisions. The same single-source-of-truth discipline for your own prices is covered in a manual price list versus one source of truth.
Product matching: why a name is not an identifier
The most expensive failure in price monitoring is not a broken collector - it is a confidently displayed comparison between two different items. "500 litre white fridge" is not an identifier:
- The same model appears at different sellers in different word orders and abbreviations.
- A retailer's internal SKU is not the manufacturer's part number.
- A twin pack, a bundle or last year's model looks almost identical in the title line.
- Colour or capacity sometimes appears only in the image or on the item page.
The answer is a stable identifier. The GS1 standard defines the GTIN key as the global identifier for a trade item, which is what a barcode encodes. Where a GTIN or a manufacturer part number exists, the match is certain. Where it does not, the item goes to a human review queue rather than into an automatic name-similarity match. A missing number beats a false one.
What you store, and what you must not republish
The distinction is simpler than it looks, and it is worth writing into the scoping document:
- Internal storage of a raw capture - so you can check afterwards whether the system read correctly. That is an internal, time-limited use.
- A derived figure - "the category median fell 4 percent this month", "we are 12 percent more expensive on three of eight items". That is what feeds a decision.
- What is not part of the method - standing up a public screen that reproduces another source's product list and prices. That is redistributing a catalogue that is not yours, and it is governed by the source's terms and the applicable law.
The practical rule: the closer the output is to a copy of the original catalogue, the higher the risk; and the more it is abstracted into an index, the more useful it is commercially. Both gradients point the same way.
Where do you get a price without reading a results page?
In order, and this ordering removes most of the work:
- A product feed from the supplier or seller. A file sent to you, or a documented interface. Stable, agreed, and with no terms question.
- A publication mandated by law. Israeli supermarket chains publish per-branch price and promotion files on dedicated transparency sites - for example Shufersal's price transparency site, which lets you pick a branch and download the files.
- A partner interface or an official commerce programme, where the source runs one.
- Documented manual entry - a person opens, screenshots and records. Slow, but legitimate, and sometimes sufficient for twenty items.
- Reading a public page - last, and only after checking that source's terms and directives.
That order is not politeness. An official feed breaks less, costs less to maintain, and does not require a discussion every time someone else redesigns their site. The comparison between routes in a shopping-engine context is worked through in collecting prices from Google Shopping.
Sources
Frequently asked questions
Is it permitted to collect prices from a comparison site?
That is not a question an article can answer. Every site has terms of use and robots directives, applicable law exists, and the answer depends on exactly what is collected and for what purpose. Run that check with a lawyer before building, not after. If the source blocks you, the answer is to stop.
What do you do when the same product shows different prices at the same seller?
Keep both observations. The difference usually comes from a promotional price against a base price, a members' price, or a second page for the same item. Collapsing the two into one value deletes the information that explains the gap, and it cannot be reconstructed afterwards.
How long should raw captures be kept?
Long enough to cover your investigation window - typically thirty to ninety days, so an odd-looking number can still be traced. Beyond that, keep only the derived figure. Retaining raw captures indefinitely grows storage cost and also grows the volume of material you are holding without a reason.
Is storing the last price per product enough?
No. Without a time series you cannot say whether a price dropped or was always that low, and you cannot spot a recurring promotion. Current state answers "what is the price" and not "what changed", and the second question is the one that justifies the project.
Keep reading
scraping
Competitor Monitoring: Treat a Website, Product, or Message Change as an Event Requiring Context
scraping
Businesses Without a Website: Build a Lead List Without Inventing Details or Bypassing Restrictions
scraping
Leads From Associations and Directories: Manage Source, Freshness, and Contact Consent
Related service
Competitor Price Monitoring
Daily competitor prices and stock, matched to your SKUs, with alerts.
About the author
Yehonatan Saadia
Freelance automation, web & MVP developer
I'm Yehonatan Saadia, a senior developer who builds business automation, custom websites, and MVPs for small and mid-sized companies across the US, Europe, and Israel. These guides come from real client work, not theory.
Work with meHave a project like this?
Tell me what you're trying to automate or build. I reply within 24 business hours with a few targeted questions, then we walk through it on a free 30-minute call, with no commitment. You come away with a scope, a timeline and a fixed price - or a straight answer that it isn't worth building.