The goal
Pull full Crexi retail CRE listings for a Washington State region with per-property detail (zoning, pricing, square footage, lease terms) — past Cloudflare and Next.js hydration.
The solution
Three-layer Python pipeline: cloudflare_utils.py solves Cloudflare via Playwright; crexi_search.py extracts assets from window.__NEXT_DATA__.props on the map-results page; crexi_properties.py drills into each asset for full details. crexi_pipeline.py orchestrates, crexi_api_probe.py was used in discovery.
Highlights
- Cloudflare bypass keeps a Playwright session alive across search + detail calls
- Hydration-aware waits eliminate empty-shell scrapes
- Reverse-engineered Crexi detail API surfaces full property metadata
The challenge
Crexi sits behind Cloudflare Turnstile — naive requests calls get 403s.
solve_cloudflare_checkbox() via Playwright completes the challenge; the browser session is reused so subsequent navigation inherits the cookies.
The search page renders listings only via __NEXT_DATA__ after Next.js hydration — scraping too early returns an empty shell.
wait_for_home_ready() polls window.__NEXT_DATA__ && window.__NEXT_DATA__.props with a 60s timeout before reading — reused in the detail fetcher too.
Search results lack full detail — zoning, financials, lease terms live only on the property page.
Two-stage pipeline: search → list of asset IDs → per-asset detail fetch; each stage writes sample JSON to keep the schema auditable.
Crexi doesn't document their internal API — schema discovery was required.
crexi_api_probe.py captured and pretty-printed raw responses during the early discovery phase and stayed in the repo as living documentation.
What was delivered
- crexi_pipeline.py orchestrator
- crexi_search.py map-results extraction
- crexi_properties.py detail fetcher
- cloudflare_utils.py bypass helper
- crexi_api_probe.py schema discovery tool
- debug_utils.py, logger.py
- Sample captures (crexi_property_sample.json, crexi_search_sample.json)
- requirements.txt pinned deps
Results
Cloudflare
Bypass
Detail API
Enrichment
Map-bounded
Coverage
What it taught me
- For Next.js sites, waiting for __NEXT_DATA__.props to hydrate is the single biggest reliability win
- Cloudflare bypass via a real browser session + cookie reuse is the simplest durable pattern
- Keeping a throwaway probe script as documentation saves hours when the schema shifts