All projects

KSP.co.il Full Catalog Scraper

curl_cffi + browser-impersonation scraper that dumps the full Israeli KSP catalog to JSON + Excel-friendly CSV.

The challenge

requests.get(...) returns empty/blocked because KSP does TLS fingerprinting and checks sec-ch-ua-* headers.

The solution

A single Python script (fetch_ksp_all.py) that replicates KSP's internal SSE catalog endpoint via curl_cffi (Chrome impersonation), paginates 50 items per page through the 15-level category path, and writes both ksp_all_items.json and an Excel-friendly UTF-8-BOM CSV. A ksp_to_csv.py post-processor converts existing JSON without re-scraping.

Results

50

Page size

JSON + CSV (UTF-8 BOM)

Outputs

KSP.co.il full catalog

Target