The goal
Carson generates legal leads from probate/foreclosure/sheriff-sale/tax-sale notices across PA. Each county has a different portal. The goal was one scraper package mapping 10+ portals into a unified lead pipeline with BatchData enrichment, SQLite storage and Google Sheets export.
The solution
A multi-portal framework: config.py holds keywords-to-category mappings, county lists, proxy/captcha/ZenRows/BatchData credentials. scrapers/ has per-portal modules (civil_core.py + public_notice.py + per-county trackers with proxy / without-proxy variants). run_test.py runs any subset. Output persists to SQLite and exports to Google Sheets via a service account.
Highlights
- 10+ PA portals unified under one lead schema
- Keyword-to-category config beats ML for this scale
- with-proxy/without-proxy variants tame flaky portals
- BatchData enrichment makes leads actionable before the sheet
The challenge
10 different portals with different layouts, search forms and pagination styles.
civil_core.py implements the shared civil-portal workflow; per-county modules subclass/extend it. public_notice.py is a dedicated single-site scraper. All scrapers normalize to the same lead schema before SQLite.
Notices arrive as free text; same notice can be probate, foreclosure or tax depending on a phrase in the body.
PUBLIC_NOTICE_KEYWORDS in config.py maps phrases to categories; adding a category is a config edit, not code.
Some portals work over residential proxies but rate-limit direct; others refuse proxy IPs.
Two variants per portal — track_a_civil_with_proxy.py and track_a_civil_without_proxy.py — each tuned for its path; operator picks per-county.
Name+address from a court notice isn't actionable — need owner, phone, property metadata.
BatchData API integration enriches each lead before SQLite, so the Sheets view is immediately usable.
Some portals use CAPTCHAs; others serve different HTML to residential IPs.
2Captcha integration + ZenRows residential rendering as configured fallbacks.
What was delivered
- scrapers/ package with per-portal modules (civil_core.py, public_notice.py, per-county trackers)
- config.py centralizing keywords, counties, proxies, APIs
- run_test.py master test runner
- SQLite storage + Google Sheets export
- BatchData + 2Captcha + ZenRows + BrightData integrations
- With-proxy and without-proxy variants
- requirements.txt
Results
10+
Portals
Resumable
Resilience
4 (probate/foreclosure/sheriff/tax)
Categories
BatchData
Enrichment
What it taught me
- Keyword-to-category config is the cheapest, most maintainable classification for a legal-notices pipeline at this scale
- Keeping with-proxy and without-proxy variants per portal turns 'sometimes breaks' into 'flip the flag'
- A single SQLite source-of-truth + separate Sheets export keeps the analyst view decoupled from raw data