The goal
Finding an apartment in Israel means checking 5+ platforms many times a day while listings disappear in minutes. The goal was a single real-time aggregator that gives users a first-mover advantage.
The solution
An end-to-end production platform: Python/Playwright scrapers for 4 Israeli sources on 3–10 minute cadences, a Node.js/Express backend with Redis+Bull async processing, OpenAI gpt-4o-mini normalization of Hebrew text, S3 image mirroring, MongoDB store, Vue 3 + Tailwind mobile-first frontend, a liveness checker, an alert engine for saved searches, and full AWS infrastructure provisioned with Terraform.
Highlights
- 4 heterogeneous sources unified through a shared base_scraper + AI normalization layer
- ~$0.002 per-listing AI cost for full Hebrew normalization
- Soft-delete + S3 image mirroring turn hostile sources into durable data
- Entire AWS stack provisioned with Terraform (EC2, ALB, CloudFront, Route53)
The challenge
Facebook aggressively detects and blocks bots, but it's the freshest source of real listings.
Session-based Playwright account manager that rotates logged-in sessions, persists cookies, limits to a single concurrent run via a dedicated thread-pool executor, and backs off on challenge pages.
Four sources with four different shapes — JSON API, GraphQL, HTML, and free-form Hebrew text.
A shared base_scraper abstract class with a fixed fetch→parse→normalize→persist lifecycle, and OpenAI gpt-4o-mini as the universal normalizer for Hebrew free text (~$0.002/listing) so downstream code is source-agnostic.
Listings expire within minutes — scrapers must run every few minutes without hitting quota/IP bans.
APScheduler with per-source executors (Facebook 10 min single-instance, Yad2/Homeless/Madlan 5 min, liveness 30 min), resumable scrapers, and BrightData proxies for Yad2. misfire_grace_time handles deploy/reboot windows.
Source image URLs expire and listings are removed before users even see them.
Download all images to S3 on first scrape and serve through CloudFront. Combined with soft delete (`isActive: false`), historical context is preserved beyond the source's lifetime.
What was delivered
- Python scrapers (Facebook, Yad2, Madlan, Homeless) with shared base class
- Node.js/Express backend with Redis+Bull async queue
- Vue 3 + Tailwind mobile-first frontend with admin dashboard
- Terraform IaC for AWS (EC2, ALB, CloudFront, S3, Route53, ACM)
- APScheduler service with per-source executors and liveness checker
- PROJECT.md architecture/vision document
Results
4
Sources integrated
3–10 min
Refresh cadence
~$0.002
AI cost per listing
Terraformed AWS
Infrastructure
What it taught me
- For Hebrew NLP on user-generated content, gpt-4o-mini hits the accuracy/cost sweet spot where regex fails
- Soft-delete + mirrored images turn transient data into a durable dataset
- A shared base_scraper class makes adding the next source a few hundred lines, not a new pipeline