Are Israeli supermarket prices publicly available?

Yes. Price transparency regulations require large food retailers to publish product prices for their stores as downloadable files, per branch, updated daily. Three file types are published per store: a stores list, an items-and-prices file including barcodes, and a promotions file. Because publication is the legal obligation, this is a published data source rather than scraping.

Why does a price comparison need the promotions file?

Because in food retail the effective price is usually the promotional one - a multi-buy, a member price, a quantity discount. Comparing shelf prices alone produces wrong conclusions, since a product that appears more expensive can be cheaper in practice. If you are building a comparison, the promotions file is not optional.

Can I compare prices at chain level?

Not meaningfully. Prices are published per branch because they genuinely differ between branches of the same chain, so averaging across a chain produces a number that exists nowhere. The useful question is what the price is at the branches within a given catchment area, not a national average - and the store-by-product multiplication is also what determines the project's size.

How do you match the same product across different chains?

By barcode, never by name - the same item appears under different spellings, abbreviations and word orders at different chains. Start with exact barcode matching and let everything else fall into a review queue rather than attempting automatic name matching, because a wrong product match is worse than a missing one: it produces a false conclusion that looks correct.

What should happen when a chain does not publish a daily file?

Alert, and do not fill the gap silently. Backfilling a missing day with the previous day's data produces a false time series that looks complete, and any trend analysis built on it is wrong in a way nobody will notice. The same applies to products that disappear - absent today does not mean delisted, it may be temporarily out of stock.

Israeli Shelf Price Data - Yehonatan Saadia

Large Israeli food retailers are required to publish prices, promotions and store lists as downloadable files, updated daily. That is a legitimate data source rather than scraping - and the engineering problem is volume and normalisation, not access.

This is one of the few cases where useful commercial data is legally and structurally available in Israel. Price transparency regulations require large food retailers to publish the prices of products in their stores - as downloadable files, per branch, updated daily. There is no terms-of-service question here, because publication is the obligation.

Why this differs from scraping

This is the point to understand first. With web scraping there is always a question - what the terms say, whether access is permitted, what happens when the site changes.

Here the publication itself is the regulatory requirement. The files are intended for download, and links to them are aggregated through public channels. That turns it from a project with risk into an ordinary engineering project - a meaningful difference when pricing for a client.

One caveat: what you are permitted to do with the data after downloading it - particularly if you build a commercial product on top - is a separate question worth putting to a lawyer. Access is permitted; commercial use is a different conversation.

What is in the files

Three file types are published per store:

File	Contents
Stores	The branch list and their identifiers
Items and prices	Product name, barcode, price, price per unit of measure
Promotions	Promotional prices and their terms

Formats include XML and Excel, usually compressed. And the update cadence is what determines the architecture: an update is required close to a change, plus a full daily file at the end of each day.

The promotions file is what most people miss

The temptation is to pull only prices and ignore promotions. That is an analytical error, because in food retail the effective price is usually the promotional one - three for a fixed amount, a member price, a quantity discount.

Comparing shelf prices alone produces wrong conclusions: a product that looks more expensive can be cheaper in practice. If you are building a price comparison, the promotions file is not optional.

The three decisions that determine whether this works

1. The unit is a store, not a chain

The first mistake is thinking "what is the price at chain X". There is no such thing. Prices are published per branch because they genuinely differ between branches.

Averaging across a chain produces a number that exists nowhere. If your client wants to know who they compete with, the question is what the price is at branches within their catchment - not a national average.

The scope implication: stores multiplied by products is the multiplication that determines project size. Worth computing before committing - it easily reaches millions of rows per day.

2. The barcode is the key

The same product appears under different names at different chains - different spelling, abbreviations, a different word order. Matching by name does not work.

The barcode is what allows like-for-like comparison. There are still edge cases to design around: products with no barcode (loose produce, weighed items), a chain's internal barcode, and different pack sizes of the same brand.

What works: start with exact barcode matching, let the rest fall into a review queue, and do not attempt automatic name matching. A wrong match between products is worse than a missing one, because it produces a false conclusion that looks correct.

3. This is volume, so it is a pipeline

A daily update per store is not a script - it is a data pipeline. What it needs:

Store the raw file as received, before processing. When a number looks odd in a month, that is the evidence of what was actually published.
Incremental processing - do not reprocess everything daily.
Idempotency - the same file processed twice does not duplicate rows.
History. The real value is not today's price but how it changed. Keep a time series, not a current state.

What breaks in practice

Dirty Hebrew. Double spaces, invisible characters, several quote forms. Two values that look identical and do not compare as strings. Normalise at ingestion.
Structure varying between chains. The requirement is uniform, the implementation less so. Expect different field names and wrap each source in its own translation layer.
A file that was not published. It happens. Do not fill the gap silently - alert. A missing day backfilled with yesterday's data produces a false time series.
Products that vanish. A product absent today is not necessarily delisted - it may be temporarily out of stock. Distinguish the two or refrain from concluding.

What it is used for

A retailer pricing against the branches in their area.
A supplier seeing how their product is priced across chains - and where it is on promotion without their knowledge.
Market research - price trends over time within a category.

In each case the expensive part is not the download. It is normalisation, barcode matching and reliability over time - roughly the ratio between a day's work and a fortnight.

When not to build it

If what is needed is a one-off look at ten products across three branches - do not build a pipeline. A manual download and a spreadsheet give the answer the same day, with no maintenance.

Building pays off when continuous tracking, a time series or broad coverage is genuinely needed. Otherwise it is infrastructure somebody will have to maintain for a question that was asked once.

Israeli Shelf Prices Are Published by Law - Here Is How to Use Them

Key takeaways