The goal
Reliably pull the latest N posts/reels from specific Facebook business profiles — surviving FB's rotating selectors, request blocking, and cookie expiry.
The solution
A Selenium-driven Python CLI (main.py) that accepts profile URL + limit and returns JSON + optional CSV. Auth is cookie-based (facebook_cookies.pkl) with email/password fallback. functions.py provides login/navigation/checkpoint-detection primitives; main.py writes outputs atomically (temp → rename) so kills mid-run never corrupt data.
Highlights
- Atomic writes mean a kill mid-run never corrupts output
- Checkpoint detection turns silent FB blocks into loud actionable errors
- Built-in DOM dumps make selector maintenance a diff away
The challenge
Facebook's DOM selectors rotate — hard-coded XPaths break within weeks.
Debug-dump flags save the current DOM so when selectors shift the operator can diff an older dump and update functions.py quickly; dumps live alongside the code.
A mid-run cookie expiry used to crash without producing partial output.
is_login_page and is_checkpoint_page are checked right after open_profile_posts; on failure the scraper exits with a clear error before touching the output file.
Killing the script while json.dump was running left the output half-written.
Write to output.json.tmp then Path.replace(output) — OS-level atomic rename means the final file is either new data or unchanged from the previous run.
Headless mode hides the DOM state that's needed to debug selector breaks.
--headless is off by default in dev; headed mode + screenshot/HTML dump on checkpoint lets any failure be reproduced offline.
What was delivered
- main.py CLI entry point
- functions.py + functions_old.py selector layer
- convert_cookies_to_json.py helper
- facebook_cookies.json schema example
- inspect_page_dump.py debug utility
- Sample output scraped_posts.json
Results
Cookies + fallback
Auth modes
JSON + CSV
Output
Atomic writes
Safety
What it taught me
- For sites with rotating selectors, built-in DOM dumps + diff-based maintenance beats any CSS-selector cleverness
- Atomic output writes are worth every line — one mid-run kill that ruins a 20-minute scrape is a lesson that sticks
- Checkpoint detection before touching the output is the biggest 'doesn't break in production' pattern