In short
A real-estate professional wanted the whole Israeli listings market as a spreadsheet instead of browsing two websites page by page. I built a tool that collects every sale and rent ad from Yad2 and Madlan, about 167,000 listings and 1,070 new-build projects, into one Excel file in about an hour and a half. The file also shows how complete it is and how accurate it is, checked against the sites themselves.
Demo video
The situation before
Yad2 and Madlan show listings one page at a time, with filters built for someone looking for a single apartment. Anyone who needs the market as a whole, to compare prices by neighbourhood, find new listings first or follow price drops, ends up copying ads by hand or working from a partial picture.
What was built
A tool that runs on demand, walks both sites across the whole country and writes every listing into an Excel file: one sheet for Yad2, one for Madlan and one for new-build projects on each site. Each row is one ad, with up to 60 columns: city, neighbourhood, street, asking price, previous price, rooms, floor, size, condition, parking, safe room, elevator, publish date, private or agent, the ad text and a link that opens the ad. Madlan rows also carry the neighbourhood scores and the estimated rent and yield the site shows. Two extra sheets report coverage (how many ads were collected against how many each site says it has) and accuracy (a random sample of ads opened on the site and compared field by field). A run that stops halfway continues from where it stopped, so a second run two weeks later costs the same and gives the same columns.
What changed for the business
- About 122,000 Yad2 listings, 45,000 Madlan listings and 1,070 new-build projects in one file
- The whole country in one run of about an hour and a half
- 99.1% to 100% of the ads each site reports, per region and deal type
- 420 of 420 checked fields matched what the ad page shows
- Filter by city, price or rooms in Excel itself: Haifa alone is 6,401 rows
This fits you if
- You are an agent, appraiser, investor or developer who needs market prices by area, not one listing at a time
- You want new listings or price drops before others see them
- You already work in Excel and want the data there, with the columns you choose
- You need the same data again every week or two, in the same format
Need something similar?
Tell me what you need, and we will check together on a short call whether and how it can be built.
Gallery
Common questions
Can I get only one city or one type of property?
Yes. The full run covers the whole country, and the same tool can run on a filter you define, such as one city, sale only, a price range or a number of rooms. The columns are agreed from a sample table before the first run.
Does the file include the advertiser's phone number?
The file has a link to every ad, which opens it on Yad2 or Madlan, and a column that says whether the ad is private or from an agent. The sample on this page has advertiser names and phone numbers removed.
How do I know the data is correct?
The file checks itself. One sheet compares the number of ads collected with the number each site shows for the same search, per region. Another opens a random sample of ads on the site and compares each field with the file. On the first run, every checked field that the ad page displays matched.
Can you collect from other websites too?
This project covers Yad2 and Madlan. I have also built collectors for Homeless and for real-estate and car sites in other countries, so another site is usually a matter of building one more collector into the same file.
Technical detailsHow it is built, for those who want to know
The solution
A resumable scraper that walks both sites country-wide: Yad2 through its own JSON endpoints (the map API split by region, price and rooms until no slice is capped, then one details call per listing), Madlan through its public sitemaps and batched GraphQL (100 listings per request). Every record lands in SQLite as it arrives, so a stopped run continues where it stopped. A separate step builds the Excel file: one sheet per source and type, the listing link in column B, a coverage sheet that compares what was collected with the total each site reports for the same search, and an accuracy sheet that opens a random sample of listings on the site itself and checks the values against the file. The first full run took about an hour and a half.
Highlights
- About 122,000 Yad2 listings, 45,000 Madlan listings and 1,070 new-build projects in a single run of about an hour and a half
- Coverage of 99.1% to 100% against the totals the sites themselves report
- 420 of 420 displayed fields matched in a random on-site accuracy check
The challenge
Yad2's map API returns at most 200 listings per request, and the country has well over 100,000.
The index splits each region by price and then by rooms until every slice is under the cap, and logs any slice that is still capped, so a gap is visible instead of silent.
Both sites run bot protection, and plain HTTP requests get a challenge page.
One persistent Chrome profile per site passes the check once; every data request then runs as a fetch inside that page with its cookies. A block triggers a growing pause and a re-warm, and repeated blocks stop the run instead of hammering the site.
A file with 170,000 rows is useless if nobody can tell how complete or how accurate it is.
Coverage compares collected listings with the total each site reports per region and deal type (99.1% to 100% on the first run). Accuracy opens a random sample on the site and compares 423 fields: all 420 that the page displays matched.
Excel treats text that starts with = as a formula and rejects control characters, and listing descriptions contain both.
A single cleaning step neutralises leading formula characters and strips illegal characters; only the listing link is written as a formula, with the URL also stored as its cached value.
What was delivered
- Resumable Python scrapers for Yad2 and Madlan
- Excel builder with coverage and accuracy sheets
- The delivered workbook, plus a sample file you can download here
Results
122,243
Yad2 listings
45,079
Madlan listings
1,072
New-build projects
420/420
Field accuracy
What it taught me
- Count against the site's own total before you trust a scrape
- Store every record on arrival so a block costs minutes, not the run
- Check accuracy on the live page, not on your own parser