Mention monitoring that measures topics rather than people. What gets stored and what does not, which channels a source offers, and why closed groups stay out.
Key takeaways
- A real business question is nearly always aggregate: is a complaint pattern forming, which term rises alongside the brand name, when did the shift begin. None of those requires knowing who wrote anything.
- A closed group, a forum requiring a login, and a private message are out of scope. If they offer no public channel, that signal simply does not exist for you, and that is a legitimate answer.
- The channels a source volunteers are the foundation: an RSS or Atom feed per the RSS 2.0 specification, a sitemap per the sitemaps.org protocol, a site's own published search, and an official API where one exists.
- The robots.txt file is read first, not last. It is part of defining the scope, not an obstacle at the end of development.
- Storing an author name, a handle or a profile link turns a topic metric into a collection of personal conversations, without being needed for any of the questions the system is supposed to answer.
Mention monitoring built to support business decisions measures topics over time, not people. The unit of record is a term, a source and a day with a counter, not a message by an identified person. Closed groups and private messages sit outside the scope, rather than inside it as a technical challenge.
This is a technical description of a measurement system, not legal advice.
What the system measures and what it never stores
That decision belongs in the schema rather than in a policy, because a schema enforces itself and a policy depends on memory. If there is no column for an author name, nobody fills one in by accident on the next pass.
| Stored | Not stored |
|---|---|
| Source domain and source type | Author name, handle or user id |
| The public page URL | Link to a personal profile |
| Publication date and sample date | Profile picture or signature |
| The matched term and the text language | Message text tied to a specific person |
| Mention count per day per source | Accumulated history per person |
| Context category at thread level | Any ranking or scoring of participants |
The left column is enough for every report anyone actually asked for. The right column is what turns a measurement system into a file on people, and once it exists in the schema somebody will build a view on top of it, and taking that apart later is hard.
Which channels does the source itself offer?
Before asking how a source could be read, ask what it offered. A site publishing a feed or a sitemap has said explicitly that it wants to be read that way.
- An RSS or Atom feed. The RSS 2.0 specification defines an item with a title, link, description and publication date, which is exactly the field set a topic metric needs.
- A sitemap under the sitemaps.org protocol, defining a URL list with
lastmodso you can identify what changed without crawling everything. - A site's own published search, exposed as a page with a parameter and intended for public use.
- An official platform interface, where one exists and its terms permit this use.
- Archive or tag pages the site publishes, which sometimes give better coverage than the feed does.
If a source offers none of these, the correct conclusion is that the source offers no machine channel. That is not a cue to find another way in. It means the source stays outside the coverage, and the coverage is documented as partial.
Why is a closed group out of scope?
Because it is closed, and that is the whole answer. A personal account you happen to be a member of is not an authorisation for a monitoring system, and the fact that reading from it is technically possible does not change the status of the source.
- A group requiring approval to join.
- A forum showing content only to signed-in users.
- Direct messages and private correspondence.
- A channel whose access is sold as a subscription.
- A source that explicitly blocks automated access.
The practical consequence is that your coverage is partial, and that is fine as long as it is declared. A report stating "eight public sources, three further sources not covered" is a report you can rely on. A report claiming to cover everything is one somebody will draw a wrong conclusion from about the size of the phenomenon.
How to build a metric worth alerting on
A raw counter makes a poor alert, because mention volume moves for reasons unrelated to the brand. The basis is a ratio against a rolling baseline, not an absolute number.
- Define exact terms, including common misspellings and both Hebrew and English forms.
- Define negative terms, to neutralise words that are both the brand name and an ordinary word.
- Count on the combination of term, source and day.
- Compute a 28-day rolling median for each such combination separately.
- Alert when a day exceeds the median by a pre-set factor, and only when the counter clears a minimum floor.
- Attach up to five public links as context, with no author details.
Step five is what prevents an alert flood. A small source rising from two mentions to six is a threefold jump and is not an event. A minimum floor on the counter filters it out without touching sensitivity on the larger sources.
What to keep, and for how long
Separating the layers is what lets you hold long history without holding a pile of text.
- Aggregate counters by term, source and day: keepable for years, because they are small and identify nobody.
- Public page URLs selected as context: a short period, because they age and a page may be deleted.
- Raw HTML snapshots: the shortest of all, and only while you need them to debug a parser.
- Term definitions and the rules version: forever, because without them an old number cannot be interpreted.
That last item is almost always forgotten. A two-year mention chart is meaningless if somebody added a term or removed a source halfway through. A rules version with a start date, stored alongside the counters, turns a definition change into a vertical line on the chart instead of a mysterious jump.
Which business question does the metric actually answer?
Not every question. It is worth knowing in advance which ones the system answers and which it does not, so expectations are not built on it that it cannot meet.
| The question | Does the aggregate metric answer it | What it needs |
|---|---|---|
| Is mention volume growing | Yes | Counters over time |
| Which sources discuss us | Yes | Breakdown by domain |
| Which topic sits next to our name | Yes | Co-occurring terms in the thread |
| When did a change begin | Yes | A rolling baseline |
| Who wrote the message | No | Deliberately out of scope |
| What happens in a closed group | No | There is no public channel |
Building the pipeline for watching external sites is covered in monitoring competitor website changes, and the messaging and product angle in monitoring competitor product and message changes. A general overview of news and brand monitoring sits in news and brand monitoring with scraping.
Sources
Frequently asked questions
Can I store a username anyway if it is public?
Technically yes, and the question is why. If no report requires it, that column only enlarges what you hold and changes the character of the system. If a report does require it, it is worth checking whether the question itself can be restated as an aggregate one.
What if most of the conversation happens in closed groups?
You accept that the metric covers a part. You can supplement it through channels the business controls, such as support enquiries and customer surveys, where the information is given directly. What you do not do is conclude that the metric represents the whole conversation, because it does not.
Is an RSS feed enough to monitor a site?
Sometimes yes and sometimes no. Many feeds carry only the most recent items or only headlines, so it is worth comparing one against the sitemap before relying on it. If the feed misses entire categories, document the gap rather than assuming full coverage.
How many negative terms do you really need?
That depends entirely on the name. A brand whose name is a common word will need a long list and a recheck every few months, while a brand with a distinctive name will need almost nothing. The way to find out is to sample fifty matches by hand and count how many are wrong.
Keep reading
Related service
Business Automation
I build custom automations that remove repetitive work end to end.
About the author
Yehonatan Saadia
Freelance automation, web & MVP developer
I'm Yehonatan Saadia, a senior developer who builds business automation, custom websites, and MVPs for small and mid-sized companies across the US, Europe, and Israel. These guides come from real client work, not theory.
Work with meHave a project like this?
Tell me what you're trying to automate or build. I reply within 24 business hours with a few targeted questions, then we walk through it on a free 30-minute call, with no commitment. You come away with a scope, a timeline and a fixed price - or a straight answer that it isn't worth building.