What is the first step in building competitor price monitoring?

Deciding what a price change should actually cause, before looking at any source. If the answer is "we'll check the dashboard", it becomes a dashboard opened twice and then abandoned. An alert to a named person on a defined threshold needs depth on a few competitors; market research needs breadth - two different architectures with very different costs.

Do I need to scrape to monitor Israeli retail prices?

Not in food retail. Price transparency regulations require large chains to publish prices, promotions and store lists as downloadable files updated daily, so if your competitors are food chains the data is already available. Generally: check for a mandated publication, API or feed before building a scraper - a published source is more stable, cheaper and carries no terms-of-service question.

Why is matching products by name dangerous?

Because a wrong match produces a confident but false number. The dashboard reports you are 15% more expensive than a competitor when in fact two different products were compared, and someone cuts a price on that basis. Match on identifiers - barcode, manufacturer part number, model - and send anything unmatched to a human review queue. A missing figure beats a false one.

Why should a price monitor alert on a drop in coverage?

Because a source that broke looks exactly like an insight. If 400 products were read yesterday and 12 today, the scraper or feed failed - but on a chart it reads as a competitor delisting products. That is the most dangerous failure mode in monitoring, since it presents as information rather than as an error, and decisions get made on it.

Should competitor monitoring update my prices automatically?

Only with a control in place, and only as a business decision that has been signed off - it is not a technical choice. A matching bug that cuts prices is direct revenue damage, so at minimum add a threshold that halts any unusually large change and routes it to a person before it takes effect.

Monitor Competitor Prices in Israel

Start with the sources that are published rather than scraped, match on identifiers rather than names, and decide what a price change should actually cause - because most price monitoring produces a dashboard nobody acts on.

"I want to know what competitors charge" is a reasonable request that usually gets built the wrong way round - starting with a scraping tool, then discovering that some data was legally available, that product matching is wrong, and that nobody acts on the dashboard.

The correct order is the reverse.

Step 1: decide what it will cause

Before looking at sources. What happens when a competitor's price drops 8%?

If the answer is "we'll look at the dashboard", it ends as a dashboard opened twice in the first week and never again. That is the most common outcome of price monitoring projects, and it is waste.

Answers that justify building:

An alert to a specific person when a defined product crosses a defined threshold.
An automatic price update by rule - a business decision requiring sign-off, not a technical choice.
A periodic report into a meeting that already happens.

The practical difference: if the answer is an alert, you need to monitor dozens of products at a few competitors. If the answer is market research, you need breadth. Two entirely different architectures - and choosing wrong here is a multiple in cost.

Step 2: exhaust the published sources

This is the step people skip, and it saves the most.

In Israeli food retail, prices are published by law - downloadable files per branch, updated daily, including promotions. If your competitors are food chains, you are done - no scraping needed at all.

The general rule: before building a scraper, check whether an API, a feed or a mandated publication exists. A published source is more stable, cheaper, and carries no terms-of-service question. Scraping is what you do when there is no alternative, not the default.

Step 3: if scraping is necessary

Outside food retail there is usually no mandated publication and you must read from websites. Three points:

Check the terms of use and robots.txt for each source. That is not a formality - it determines whether the project is legitimate. And for commercial use, that is a question for a lawyer rather than a developer.
Scrape at a human pace. Not only to avoid blocks - another business's server is not your resource.
Expect it to break. Sites change. Maintenance is part of the cost, not a one-off incident.

The caveat worth telling a client up front: scraping is an ongoing cost, not a one-off project. Anyone pricing it as a project is closing at a loss.

The problem that decides everything: product matching

This is the real work, not the fetching.

A product name is not an identifier. The same item appears at competitors under different spellings, abbreviations and word orders. Comparing by name produces wrong matches.

And what makes that dangerous: a wrong match produces a confident, wrong number. The dashboard shows you are 15% more expensive than a competitor - when in fact two different products were compared. Someone will cut a price on the strength of it.

The rule: match on an identifier - barcode, manufacturer part number, model. Anything not matched confidently goes to a human review queue. No automatic name matching. A missing figure beats a false one.

The listed price is not the price paid

A second mistake that makes the analysis misleading. What changes the effective price:

Promotions and quantity discounts - in retail, usually the real price.
Shipping - a cheap product with expensive delivery is not cheap.
Member pricing versus the general price.
VAT - confirm you are comparing on the same basis.

If you are comparing base prices only, say so explicitly in the report rather than letting the number look definitive.

Architecture

Keep a time series, not a current state. The value is not "what is the price" but "what changed". Without history there is no monitoring, only a snapshot.
Keep the raw data. When a number looks odd, that is the evidence of what was actually read.
Alert on a sharp change in coverage. If 400 products were read yesterday and 12 today, the source broke - and that must not look like "the competitor delisted products". This is the most dangerous failure mode in monitoring, because it presents as an insight.
Idempotency. A repeated run does not duplicate rows.

What not to do

Do not monitor everything. Fifty products that genuinely matter beat five thousand nobody looks at - and cost far less to maintain.
Do not auto-update prices without a control. A matching bug that cuts prices is direct revenue damage. At minimum, a threshold that halts an unusual change.
Do not treat scraping as a closed project. It will keep requiring work.

Checklist

Decide what a price change triggers - before choosing a source.
Check for a mandated publication, API or feed. In food retail there is one.
Check terms of use for each remaining source; commercial use goes to a lawyer.
Match on identifiers only; anything unmatched goes to a review queue.
Include promotions and shipping, or state explicitly that you did not.
Keep a time series and the raw data.
Alert on a coverage collapse rather than interpreting it as an insight.

Monitoring Competitor Prices in Israel: Sources, Order and What Breaks

Key takeaways