How to Scope a Web Scraping Project: What to Prepare Before Asking for a Quote

The difference between a scraping project that flows and one that drags is nearly always decided in the brief. Six inputs - sources, fields, volume, frequency, delivery format and purpose - turn a vague estimate into an accurate quote, and usually a cheaper one.

Six inputs that turn a brief into a quote

The first row saves the most time in conversation. One site can hold five page types with five different structures, so "scrape the site" is not a specification - three representative URLs are.

The sample row: the most useful document in a brief

Before any prose description, open a spreadsheet and fill in one row by hand, exactly as it should look when the project is finished. A column name per field, and a real value in each cell.

That does three things at once. First, it exposes fields you thought of as one that split into two - "price" turning out to be price, currency, and price before discount. Second, it reveals fields that do not exist on the site at all, and that is a surprise best delivered on day one. Third, it becomes the acceptance criterion: the project is done when the file looks like that row, ten thousand times over.

What actually drives the price

Counter-intuitively, the number of fields barely matters. The three things that do:

• Frequency - a one-off run is a script; an hourly run is a system with monitoring, alerting and failure handling.
• The site's defences - a site serving static HTML and a site with rendering, rate limiting or a human challenge are two different worlds.
• The completeness requirement - "most of the records" and "every record, verified" are not the same project.

The third is the one businesses do not think about and later argue over. If two records in a thousand are missing and that does not matter, it is one project; if every missing record is a problem, you need verification, a comparison against a declared count, and exception handling - and that changes the price more than any field you add.

Why you are asked about the purpose

Not out of curiosity. The purpose decides three things that cannot be inferred from the fields: whether an official source exists that would be cheaper and more stable, whether personal data is involved, and whether the frequency you asked for is genuinely needed.

The most common case is the first. A client asks to scrape a platform that has a public API, because they did not know there was one. The full comparison is in web scraping against an API, and the simple rule is that an official interface wins nearly every time - it is stable, faster, and does not break on a redesign. This is what I do for clients - see web scraping services.

The second case, personal data, is where the answer may be not to proceed at all. The factual background is in is web scraping legal in Israel; the decision itself belongs to a lawyer, not to a supplier.

What to check before committing

• That the source does not require a login to reach the data.
• That what you see in the browser also exists without running JavaScript, or that it is known not to.
• That there is a way to identify a record uniquely and stably - an id, not a title.
• That pagination is predictable rather than infinite scroll only.
• That the site does not change content by country or by logged-in user.
• That the historical depth is known - whether you also need the past.

The third item decides whether data can be updated or only re-collected every time. Without a stable identifier, every run is a fresh set and there is no way to know what changed - and that is precisely the capability most clients assume exists.

What a healthy process looks like after the brief

The order that works is a short feasibility check before a binding quote: pulling twenty records from the real source. That takes a few hours and answers the question no description can - what the defences actually are, and which fields are genuinely reachable.

A quote given without that check rests on assumptions, so it is either padded to cover risk or precise-looking and revised halfway through. Both outcomes are bad, and the pre-check prevents both.

How long does collecting the brief take?

At a business that knows what it wants, an hour - most of it on the sample row, which is exactly the time that pays. When it takes longer, the reason is nearly always that the purpose itself is not yet clear, and that is a question worth closing before ordering data.

What does not work is ordering an "initial survey" as a substitute for a brief. Scraping delivers exactly what you asked for; it does not tell you what you should have asked for.

Sources

• robots.txt - the Robots Exclusion Protocol
• Web scraping against an API
• Is web scraping legal in Israel
• How much web scraping costs
• Book a call