The goal
The client runs a regional classifieds aggregator and needed continuous, schema-unified feeds of real estate and cars from Chile, Mexico, Puerto Rico and Spain across sites with very different anti-bot stances.
The solution
A portfolio of 8 Python scrapers, one per site, all producing data that slots into two shared JSON schemas. Scrapers are split between pure-HTTP (curl_cffi + httpx), Playwright for Angular-rendered sites, and session-based browser flows for auth-gated ones, with 2Captcha wired in where needed and resumable progress checkpoints on every scraper.
Highlights
- 4 countries, 8 sites, 2 unified schemas
- Three-tier scraping architecture (httpx → curl_cffi → Playwright)
- Resumable runs on every scraper via JSON checkpoints
- Angular client-side pagination handled with Playwright + JSON query payloads
The challenge
Each of 8 sites returns different fields (bedrooms vs habitaciones, area in m² vs ft², currencies DOP/CLP/MXN/EUR) — unifying them is fragile.
A single build_record() per scraper normalizes to the shared schema: unit-converts area (m² → ft²), hashes listing IDs into a deterministic hid, maps for_sale/rent to listing_type, preserves source_url and source.
Anti-bot stances vary dramatically: Idealista/Inmuebles24 fingerprint aggressively, Chileautos/Kavak use Cloudflare, FindItPR is Angular-only, Coches rate-limits.
Three-tier approach — httpx where possible (fastest), curl_cffi with browser TLS fingerprints when blocked, Playwright + persistent sessions + 2Captcha for hostile sites. Each scraper uses the minimum complexity needed.
FindItPR builds listing URLs from a JSON filter blob (busqueda) and only renders after Angular hydration — static HTTP returns an empty shell.
Playwright sync API, build the busqueda payload programmatically with json.dumps + URL encoding, wait for listing cards to render, parse with BeautifulSoup, click through pages with delays — handles up to 600 results at 12/page.
Long runs (Inmuebles24, Idealista) take hours — a single mid-run failure throws away everything.
Each scraper writes *_progress*.json after every page (current page, collected IDs, last URL). On restart the scraper reads this file and skips completed work; combined with atomic JSON appends to the output file, full recovery is free.
What was delivered
- 8 Python scrapers organized by country (Chile, Mexico, Puerto Rico, Spain)
- Two unified JSON schemas (real_estate_object_example.json, cars_estate_object_example.json)
- Per-site config files for filters/regions
- Persistent browser session files for auth-gated sites
- requirements.txt with pinned dependencies
Results
4
Countries
8
Sites
Real estate + vehicles
Categories
Resumable
Recovery
What it taught me
- Picking the minimum scraping tier for each site (HTTP → curl_cffi → Playwright) dramatically improves throughput and reliability
- Schema unification done once per-scraper is cheaper than normalizing later
- Resume-from-checkpoint should be default — nothing burns time faster than rerunning a full crawl