The goal
Deliver a complete, one-row-per-charge dataset of every offender in the Georgia DOC Offender Query — despite the site's hard 150-record cap that silently drops ~85% of large-prison data.
The solution
A Python/Playwright scraper walking all 323 GA institutions with adaptive partitioning (subdivide by last-name first-letter until under 150), outputting 18 columns/charge to offenders.csv, with progress.json for resume and rebuild_progress.py for recovery.
Highlights
- Guaranteed 100% coverage despite a hard 150-record cap
- 50K+ charge rows across 323 institutions
- Resumable + recoverable via progress.json + rebuild utility
The challenge
The site caps results at 150 per query, silently dropping ~85% of a large institution's data.
Adaptive partitioning — start with a broad query, subdivide by last-name first-letter when results hit ~150, recurse deeper if a partition is still capped. Every completed partition recorded in progress.json.
GDC uses JSP sessions and returns 500 errors to plain HTTP clients.
Playwright runs a real Chromium — handles session cookies automatically. Visible-browser mode further avoids headless detection.
A network hiccup after hours of scraping could lose everything.
progress.json records completed partitions after every batch; rebuild_progress.py walks the existing CSV to regenerate progress if it desyncs.
Clients wanted one row per charge, but GDC shows charges as a nested table under the inmate.
Parser duplicates inmate attributes (AIS, Name, Race, etc.) across each charge row so downstream CSV consumers don't need a join.
What was delivered
- scraper.py runner with --test flag
- collector.py, parser.py, browser.py, fetch_requests.py
- progress.py + rebuild_progress.py recovery utilities
- analyze.py pre-delivery quality report
- config.py, log_setup.py
- requirements.txt + Playwright Chromium install
- README documenting workflow, schema and partitioning rationale
Results
50K+
Records
323
Institutions
100%
Coverage
Resumable
Recovery
What it taught me
- Adaptive partitioning is the canonical technique for APIs/sites with a hard result cap
- For JSP/ASP.NET government sites, Playwright is almost always faster than reverse-engineering the session
- Store progress after every partition, not every row — tiny overhead, clean recovery