Most scraping projects fail after delivery, not during it - the scraper works for six weeks and then quietly breaks. Here is how to brief the work, what separates a scraper from a data pipeline, and the questions that expose the difference before you pay.
Key takeaways
- A scraper is not a deliverable, it is a subscription. Sites change, and the real question is not whether it works today but who fixes it in month four.
- The cheapest quotes almost always assume the site is static HTML with no anti-bot measures. Ask the candidate to check the target before quoting - the answer tells you a lot.
- Insist on monitoring and alerting as part of the scope. A scraper that returns zero rows silently is worse than one that crashes, because you keep making decisions on stale data.
- Ask how they handle legality and terms of service. A developer who has never thought about it is a risk you are absorbing, not one they are carrying.
The typical web scraping engagement goes like this. You describe the site, someone quotes a low number and a short timeline, the script is delivered and it works. Six weeks later the site changes a class name, the scraper returns zero rows, nobody notices for two weeks, and the reports everyone has been reading are based on nothing.
The problem is almost never that the developer could not write a scraper. It is that a scraper was the wrong deliverable. This guide covers how to brief the work so you get something that survives, and what to ask to find out whether the person quoting understands that.
Decide What You Are Actually Buying
Three very different things get called "scraping":
| Type | What it is | What it needs |
|---|---|---|
| One-off extraction | Pull a dataset once, hand over a CSV | Just correctness. Maintenance is irrelevant. |
| Scheduled collection | Same source, repeatedly, into a store | Monitoring, alerting, retries, someone on call |
| Monitored pipeline | Several sources, cleaned, deduplicated, feeding a system | All of the above plus data quality checks and schema management |
If you brief the second or third as if it were the first, you will get the first - and it will fail exactly as described above. Say which one you want in the brief.
What to Put in the Brief
- The exact URLs, including an example of a page with the data you want. Not "product pages" - an actual link.
- The fields you need, with an example value for each. Ambiguity here becomes rework.
- Volume. A hundred pages and a hundred thousand pages are different architectures, not different runtimes.
- Frequency. Once, daily, hourly, or on demand.
- Where the data goes. A CSV, a Google Sheet, a database, an API. This is often more work than the extraction.
- What happens when a page is missing or a field is empty. Skip, flag, or fail? Decide before, not after.
- How stale is too stale. This is what defines your alerting threshold.
The Questions That Separate Candidates
"Have you looked at the target site yet?" Anyone quoting before checking is guessing. The presence of JavaScript rendering, anti-bot protection, rate limiting or login walls changes the cost by an order of magnitude. A good answer describes what they found.
"What happens when the site changes its layout?" You are listening for: monitoring that detects it, alerting that tells someone, and a stated arrangement for who fixes it and on what terms. "I'll take a look if it breaks" is not an arrangement.
"How will I know the data is wrong rather than absent?" The good answer involves validation - expected row counts, required fields, sanity ranges - not just a success or failure exit code.
"How do you handle blocking?" Reasonable answers involve respectful rate limiting, proper session handling, and proxies where appropriate. Answers that centre entirely on evasion tooling suggest the target may be a poor fit for a sustainable long-term pipeline.
"What about the site's terms of service?" You want someone who raises it themselves. The legal position varies by jurisdiction, by what is collected, and by whether personal data is involved - it is a real question with real answers, and a developer who has never considered it is transferring the risk to you silently.
"What do I get if you disappear?" Source code in your repository, documented setup, credentials you own, and infrastructure in your accounts. Not a script running on their laptop.
Red Flags
- A quote given without looking at the target.
- Maintenance not mentioned until you ask.
- Infrastructure and credentials held in their accounts.
- No mention of what happens on failure.
- An estimate that assumes every page is well-formed.
- Reluctance to discuss the legal dimension at all.
What It Should Cost
Broad ranges for a senior freelance engineer:
| Scope | Typical |
|---|---|
| One-off pull, single source, cleaned and delivered | Days |
| Scheduled collection with monitoring and alerting | 2-5 weeks plus a monthly maintenance arrangement |
| Multi-source pipeline with deduplication and quality checks | 6-12 weeks plus monthly |
A quote far below these usually means monitoring and maintenance are not in scope. That can be the right choice for a one-off. It is the wrong choice for anything you plan to depend on.
The Maintenance Conversation
Have it before signing, not after the first break. Agree three things explicitly: how quickly a break will be detected, how quickly it will be fixed, and how that fix is paid for. A modest monthly retainer that covers breakage is almost always cheaper than emergency fixes billed hourly, and it aligns incentives - the person who wrote it has a reason to write it robustly.
If you want an assessment of your specific target before committing to anything, book a free call. Related: how to scope a web scraping project, why scrapers break and what maintenance involves, and what web scraping costs.
Frequently asked questions
What should I include in a web scraping brief?
Include the exact URLs with a real example page, the specific fields you need with an example value for each, the expected volume, the frequency, where the data should end up, what should happen when a page is missing or a field is empty, and how stale the data can get before it stops being useful. That last point is what defines the alerting threshold, and it is the one most often left out.
Why do web scrapers break so often?
Because they depend on the structure of a page that someone else controls and changes without telling you. A renamed CSS class, a new consent dialog, a switch to client-side rendering, or a change in pagination will all break extraction. This is normal and unavoidable, which is why the important question in hiring is not whether the scraper works on delivery but whether breakage is detected quickly and who is responsible for fixing it.
Should the developer host the scraper or should I?
You should. The source code belongs in a repository you own, the infrastructure should run in your cloud accounts, and any credentials or proxy subscriptions should be in your name. A scraper running on a developer's personal machine or in their account is a dependency you cannot manage and cannot recover if the relationship ends. Agreeing this up front costs nothing; discovering it later is expensive.
Is web scraping legal?
It depends on the jurisdiction, on what is being collected, on whether personal data is involved, and on the target site's terms. It is a genuine question with genuine answers rather than a blanket yes or no, and it is not something a developer or a blog post can settle for your situation - a lawyer familiar with the relevant jurisdiction can. What you should look for when hiring is a developer who raises the question themselves, because one who has never considered it is transferring the risk to you without saying so.
How much should ongoing maintenance cost?
A modest monthly retainer covering breakage is the usual arrangement, and it is almost always cheaper than paying for emergency fixes hourly. It also aligns incentives, because the person who wrote the scraper has a direct reason to make it robust rather than merely working on delivery day. Agree three things explicitly before signing: how quickly a break will be detected, how quickly it will be fixed, and how that fix is paid for.
Keep reading
Related service
Web Scraping
Reliable web scraping and data pipelines that deliver clean data.
About the author
Yehonatan Saadia
Freelance automation, web & MVP engineer
I'm Yehonatan Saadia, a senior engineer who builds business automation, custom websites, and MVPs for small and mid-sized companies across the US, Europe, and Israel. These guides come from real client work, not theory.
Work with meHave a project like this?
Tell me what you're trying to automate or build and I'll tell you the fastest reliable way to ship it.