The goal
A repeatable Python tool that pulls profile data, posts and Reels from any Instagram URL without launching a browser — structured JSON output, full pagination with cookies, minimum dependencies.
The solution
A single-CLI Python script that detects URL type and follows the correct API chain: for profiles, seed the session via /username/ HTML to extract LSD+cookies, then call web_profile_info/ (REST) with PolarisProfilePageContentQuery (GraphQL) as 429 fallback. For posts/reels, PolarisProfilePostsQuery with cursor pagination. For individual posts/reels, raw HTML + no-auth oEmbed. Cookie input via string or JSON for >12 pagination.
Highlights
- Four URL types handled by a single CLI
- REST → GraphQL fallback eliminates 429 flakes
- Cookie auth unlocks full pagination
- No browser — pure requests, dramatically faster than Playwright
The challenge
Instagram's internal GraphQL rejects requests that don't look exactly like browser traffic — missing X-FB-LSD, wrong sec-ch-ua, stale doc_id all fail.
Captured the browser's exact request in DevTools, extracted APP_ID, BLOKS_VERSION_ID, DOC_ID into constants; a seed GET on /username/ pulls LSD + cookies via regex so every subsequent call validates.
The public REST endpoint returns 429 intermittently, especially without cookies.
Two-tier fetch — REST first, GraphQL PolarisProfilePageContentQuery as automatic fallback on 429.
The unauthenticated GraphQL posts query returns only the first 12 items — has_next_page is immediately false.
Cookie input via --cookies (raw string) or --cookies-file (JSON); the session attaches to every request, enabling real end_cursor → has_next_page pagination. README documents the DevTools extraction steps.
Individual post/reel URLs have a different shape — the GraphQL posts query doesn't apply.
For /p/CODE/ and /reel/CODE/, save raw HTML (for downstream extraction) and call /api/v1/oembed/?url=... for canonical metadata (author, thumbnail, media_id). Two outputs per URL.
What was delivered
- fetch_instagram_profile.py single CLI
- requirements.txt (minimal deps)
- Reference captures (web_profile_info, fetch_query, posts_fetch, reels_fetch) + sample responses
- README documenting cookie extraction and API chain
- Sample outputs (yarden.sha_profile.json, yarden.sha_posts.json)
Results
4
URL types
Anonymous / Cookies
Auth modes
REST → GraphQL
Fallback
Headless (requests)
Runtime
What it taught me
- For Instagram / Facebook APIs, capturing the browser's exact request is faster than reverse-engineering
- Two-tier REST → GraphQL fallback turns intermittent 429s into a non-issue
- Headless requests is dramatically faster than Playwright when the APIs are usable