The goal
Dump the full KSP.co.il catalog (one of Israel's largest consumer-electronics retailers) to JSON + CSV for price tracking and analysis, past their TLS-fingerprint defenses and deeply-nested category tree.
The solution
A single Python script (fetch_ksp_all.py) that replicates KSP's internal SSE catalog endpoint via curl_cffi (Chrome impersonation), paginates 50 items per page through the 15-level category path, and writes both ksp_all_items.json and an Excel-friendly UTF-8-BOM CSV. A ksp_to_csv.py post-processor converts existing JSON without re-scraping.
Highlights
- curl_cffi impersonation is dramatically more reliable than header juggling when sites fingerprint TLS
- Dual JSON + Excel-friendly CSV makes downstream analysis painless
- Captured browser reference (test_ksp.js) is the fastest way to diff against site changes
The challenge
requests.get(...) returns empty/blocked because KSP does TLS fingerprinting and checks sec-ch-ua-* headers.
curl_cffi with chrome impersonation + exact header set captured from DevTools — every header matches a real Chrome 147 request.
KSP's endpoint streams text/event-stream chunks; parsing naively as JSON fails.
Request with accept: text/event-stream, accumulate chunks, parse the combined stream at the end of each page.
The 15-level category path is fragile — getting it wrong returns empty.
Captured the exact chain from DevTools Network tab and hard-coded as CATEGORY; changing it only requires updating one constant.
Excel opens UTF-8 CSVs inconsistently with Hebrew.
CSV writer explicitly encodes UTF-8 with BOM so Excel recognizes Hebrew on open.
What was delivered
- fetch_ksp_all.py main scraper with dual output
- ksp_to_csv.py JSON-to-CSV post-processor
- test_ksp.{py,js,json,csv} browser-captured reference request + sample response
- requirements.txt
Results
50
Page size
JSON + CSV (UTF-8 BOM)
Outputs
KSP.co.il full catalog
Target
What it taught me
- curl_cffi's browser impersonation is dramatically more reliable than juggling headers by hand when TLS fingerprinting is in play
- Always emit UTF-8-BOM CSV when Hebrew data is going into Excel
- Keep a test_{site}.js capture next to the scraper — the diff against that file is your first clue when the site shifts