The goal
Deliver a complete, repeatable download of the Washington DC DLCP public business registry — a dataset not offered as bulk export.
The solution
A single-CLI Python scraper (washingtondc_pagination.py) that calls DLCP's paginationadvance-search API directly with exact headers (x-app-route, referer), % wildcard business-name search, token-based pagination, minimal record extraction, and tagged timestamped logs. --start-page enables mid-run resume.
Highlights
- No browser — plain requests beats Selenium for JSON APIs
- % wildcard unlocks the full registry in one pass
- Tagged logs make tailing runs remotely trivial
The challenge
DLCP offers the registry only through a paginated search UI — no CSV, no 'download all' button.
Reverse-engineered the paginated API call from the browser's Network tab, replayed with requests.Session, used the % wildcard to match all businesses.
API silently 403s or returns empty if headers don't match the frontend exactly.
_build_headers() mirrors the exact header set the browser sends (x-app-route, referer, cache/pragma no-cache, correct content-type).
Registry exports span hundreds of pages — mid-run failures shouldn't restart everything.
--start-page N resumes from the last successfully written page; streaming JSON output ensures a partial run is never wasted.
What was delivered
- washingtondc_pagination.py single-CLI scraper
- Sample CSV output (va-test.csv)
- run_artifacts/ structure for JSON outputs
Results
DC DLCP registry
Target
--start-page
Resume
Streaming JSON
Output
What it taught me
- For government registries, 'no bulk export' is often a headers/wildcard problem in disguise
- CLI resume (--start-page) is cheaper than building a resume DB
- A tagged logger is disproportionately useful when tailing remotely