Scraping a Competitor Website: The Legal and Technical Questions to Answer Before Collecting One Line

This is a list of questions, not answers. Before the first request to a competitor's site, four need answering: what exactly will be collected, what the site's terms of use say, what robots directives it publishes, and whether the data contains details about people. The answer for a specific site belongs to you and your lawyer.

The questions to answer before the first request

Write the answers in one document. If any of them is blank, the project is not yet defined:

• What exactly is collected? An explicit field list, not "whatever is on the page".
• What is it used for? Internal analysis, a pricing decision, a product you sell, or republication. Those are entirely different cases.
• What do the site's terms of use say about automated access, and was explicit agreement required to reach the content at all.
• What directives does the site publish in its robots file, and what do they say about the paths you care about.
• Does the collected content include protected creative work, rather than only facts such as a price or availability.
• Does the data contain anything that identifies a person, even where it is publicly displayed.
• How long do you retain it, and who deletes it.
• Who on your side signs off that the answers above were obtained, and when they get revisited.

These are not bureaucracy. They are what lets you answer calmly if someone asks, and they shorten the project, because most projects shrink dramatically the moment somebody writes an explicit field list.

What does the robots file actually specify?

RFC 9309 describes the robots exclusion protocol, and it is a technical standard rather than a legal one. Several points change the implementation:

• A crawler picks its own product token. The standard requires it to contain only letters, underscores and hyphens, and says it should appear as a substring of the identification string the crawler sends, which for HTTP is the User-Agent header.
• Matching a token against groups in the file is case-insensitive, and when several groups match they are merged into one.
• If no group matches, the crawler must obey the * group if one is present. If that is absent too, no rules apply.
• A server status in the 400 to 499 range for robots.txt means the file is unavailable, and the crawler may then access resources on the server.
• A status in the 500 to 599 range means the file is undefined, and the crawler must assume a complete disallow. That is the distinction people get backwards: a server-side failure is not permission.
• The file may be cached, but not used from cache for more than 24 hours unless it is unreachable.
• A parser must handle at least 500 kibibytes, and a crawler should follow at least five consecutive redirects.

Robots directives do not govern what you may do with data once it is collected. They state which paths crawlers are asked not to fetch. Those are two separate questions, and conflating them is the most common confusion in this field.

How do you request without harming a server that is not yours?

This is the central technical question, and it has measurable answers:

1. Send a User-Agent that identifies who you are, with a way to reach you. Anyone looking at unidentified traffic assumes the worst, reasonably.
2. Prefer an official feed or a documented interface over reading a page. It is more stable and cheaper for both sides.
3. Rate-limit, and treat a 429 as RFC 6585 defines it: it indicates too many requests were sent in a given amount of time, and it may include a Retry-After header saying how long to wait. Wait that long rather than retrying immediately.
4. Use conditional requests and caching as set out in RFC 9111 so you do not re-fetch content you already hold. Fewer requests mean less load and less chance of being blocked.
5. Run at low-traffic hours, at a rate you would be willing to defend out loud in a conversation.
6. Stop when you are blocked, and look at why. A block is an answer, not a fault to route around.

The first step on that list is also the cheapest: check whether an official interface exists at all. The full ordering between an interface and page reading is in scraping versus an API, and how to bound the project itself is in how to scope a web scraping project.

Who answers each question, and what gets recorded

That table is also the escalation line. A row whose answer is beyond the authority of whoever runs the project moves in one line to whoever does hold that authority, carrying the date an answer is needed.

What you record about every row you collect

A data source without provenance is not data, it is hearsay. For every stored record:

• The exact source address it was read from, not the name of the site.
• The retrieval timestamp, on one consistent clock.
• The response code actually received.
• The identifier of the run that collected it.
• A stored raw capture, or its identifier, for a later dispute.
• The rule or field the value was extracted from, so that when the page structure changes you can see what moved.

That metadata is what lets you answer "where did this value come from" without going back to the site. It is also what turns a complaint about a wrong figure into a one-minute check rather than an argument.

Sources

• RFC 9309 - Robots Exclusion Protocol
• RFC 6585 - Additional HTTP Status Codes
• RFC 9111 - HTTP Caching