Collecting Yad2 Listing Data: What to Settle Before You Build

Yad2 is Israel's largest classifieds board, which makes it the place people turn to when they want to understand a market - second-hand car prices, rents in an area, trends in a category. Before building anything, there is one gate to pass, and it is not technical.

The first gate: terms of use

I put this first rather than last, because that is the right order.

Classified sites have terms of use, and they usually address automated collection explicitly. Reading them is not a checklist item - it determines whether there is a project here at all.

Three practical points:

1. Read the terms of use and robots.txt for the site. They are the source, not an article.
2. Commercial use is a question for a lawyer. If you are building a product, a service or a report that is sold - the difference between personal research and a commercial product is exactly the difference that matters, and I cannot answer it in an article.
3. If there is an official route, use it. A partnership, a feed or a data agreement beats any technical solution, and it is more stable too.

I am not providing code or circumvention techniques here, deliberately. What is useful is what determines whether this works, and what the alternatives are - and that is what remains.

The second gate: personal data

And this is a gate many people do not see at all.

A classified listing is not only a price. It carries a name, a phone number, sometimes a precise location - which is personal data about a private individual.

The distinction that matters:

Collecting and storing private individuals' contact details - and certainly using them to make contact - is a privacy matter rather than a technical one. In Israel this is a regulated area, and the answer to "what is permitted" belongs with a lawyer.

My practical recommendation: if the need is market research, do not collect contact details at all. Not because it is complicated - but because they are unnecessary to the question, and they are what turns a harmless dataset into a personal database you must protect.

If you passed both gates - what determines whether this works

This is an ongoing cost, not a project

This is the most commonly mispriced point. Classified sites change structure - design, field names, how content loads. A collector that worked for three months breaks without warning.

And not only breaks - breaks silently. The job runs, does not crash, and returns zero listings or empty fields. Without an alert you discover it in a report a month later.

What must be in place:

• An alert on a collapse in coverage. If 800 listings were collected yesterday and 6 today, that is not "the market dried up", it is a broken collector. This is the most dangerous failure mode, because it masquerades as an insight.
• Structure validation on every run, stopping when something changed.
• Keeping the raw data - when a number looks odd, that is the evidence.

Maintenance is part of the cost. Anyone pricing this as a one-off is closing at a loss.

Pace

Collect at a human pace. Not only to avoid blocks - another site's server is not your resource, and an aggressive pace is also what turns a technical question into an entirely different one.

The data is dirty

Listings are written by people, and it shows:

• Price in a text field - with commas, a currency symbol, or "negotiable" and "offers invited".
• Inconsistent area - a neighbourhood, a city, or a region.
• Duplicate listings - the same property posted several times.
• Hebrew with double spaces and invisible characters.

A normalisation layer is not a luxury, and it is usually more work than the collection itself.

What to design for: trends, not listings

And this is the decision that determines whether the output is worth anything.

An individual listing disappears - it sells, it is removed, it is updated. Tracking a specific ad is a perishable data point.

What survives is the aggregate:

So: keep a time series rather than a current state. The value is not today's price - it is the change. And an aggregate time series also does not require you to store contact details.

The alternatives most people skip

Before building, worth checking:

1. Government open data. For property transactions, for instance, official sources are published for public use - see the data.gov.il guide. An official figure beats a listing, because a listing is an asking price rather than a transaction price.
2. Sources published under obligation. Such as retail shelf prices - where there is no terms-of-service question at all.
3. A direct agreement. If the need is commercial and ongoing, approach the site. It sounds unlikely and sometimes works, and it is incomparably more stable.
4. A data vendor. Some sell ready-made market data. Cost against operational quiet.

The important note: a listing is an asking price. If the question is "what did it sell for", a classifieds board is the wrong source - even if it is the most available one.

When not to build

If the question is one-off - "what is the range in this area right now" - build nothing. Half an hour of manual looking gives a better answer, with no maintenance and no legal gate.

Building pays off only when continuous tracking and a time series are genuinely needed. Otherwise it is infrastructure somebody maintains for a question asked once.

Checklist

1. Read the terms of use and robots.txt - before anything.
2. Commercial use - to a lawyer.
3. Do not collect contact details if the question is about a market.
4. Check whether an official or mandated-publication source exists.
5. Budget maintenance, not a project.
6. Alert on a coverage collapse - the failure that masquerades as an insight.
7. An aggregate time series, not listing tracking.