Repetition Is Not Knowledge

Chatterbox is the website chat product I am building for small businesses. It turns a company's website, FAQs, and documents into grounded answers for visitors, and collects contact details when the conversation needs a person. Those answers depend on knowledge that is current, distinct, and worth trusting.

When one business joined Chatterbox, its website produced more than 5,400 question-and-answer pairs. The crawler read more than 1,000 pages, increasing the time and compute needed to ingest the site.

The business did not have 5,400 useful things to say. Its website repeated similar material across pages aimed at different cities. A service description for one city became another page for the next city, with the location changed and most of the surrounding copy left alone. Old posts added another layer of facts that had once been true, or useful for search, but no longer represented what a customer needed to know.

The crawler had read the website correctly. It had understood the knowledge base badly.

The web Chatterbox has to read

In the part of the Indian SMB market Chatterbox is built for, I see this problem most often on sites maintained by small agencies working directly with local business owners.

Their clients often have small budgets, local competition, and an immediate need for leads. The agency needs a method it can repeat across many towns, neighbourhoods, and services. One low-quality SEO tactic is to publish a family of near-identical location pages or posts, changing the city name and a few local terms on each one.

I understand the incentive. The business wants search traffic. The agency has to show visible work within a limited budget. A reusable page is cheaper than researching and writing genuinely local material for every place. I do not need to judge the people doing it to address what the method leaves behind.

The publisher may see those pages as separate opportunities to rank. A knowledge system sees something else: the same claim repeated many times, small wording changes that defeat exact duplicate checks, and occasional local details mixed into the repetition.

The crawler has to work with the website that exists. Requiring every small business to repair its content strategy before it can use Chatterbox would make the product useless to the businesses it is meant to serve.

Repetition damages retrieval

The old ingestion path treated every page as an independent document. It extracted customer-facing facts, turned them into FAQ pairs, embedded them, and added them to the tenant's searchable knowledge.

In this article, a fact means a stored piece of business knowledge. The extraction step can express one fact as one or more question-and-answer pairs for retrieval.

That works on a small, carefully maintained website. It behaves differently when a site contains fifty versions of the same service page.

Questions such as “Do you provide this service in Chandigarh?”, “Do you provide this service in Mohali?”, and “Do you provide this service in Panchkula?” are not exact duplicates. Their embeddings are close, but each page can still generate several variations. A few templates can become thousands of stored pairs.

The extra pairs create three problems.

First, extraction and embedding cost grows with the page count. The system pays repeatedly to learn almost the same thing.

Second, retrieval gets crowded. Several weak variations can compete with one clear answer when a visitor asks a question. A larger corpus is not automatically a better corpus.

Third, stale material gains another chance to surface. A dated offer, old price, discontinued service, or past event can be well written and easy to retrieve. Grounding the answer in that page makes the response faithful to the source and still wrong for today.

I wrote about Chatterbox's refusal boundary in Why Chatterbox Refuses to Guess. That boundary depends on the supplied knowledge being worth trusting. Retrieval cannot rescue a corpus full of repetition and old claims.

Deleting every template would lose real facts

The obvious fixes were too blunt.

Exact text deduplication misses pages where a city, heading, or sentence has changed. Ignoring every URL that looks like a location page can remove a branch address, a local phone number, an area-specific price, or a place where the service is genuinely unavailable.

Sending every pair of facts to a language model would be expensive. The sites most in need of cleanup are the same sites that produce the largest comparison set. Paying for a pairwise comparison after every crawl would turn a quality fix into another source of unpredictable ingestion cost.

The crawler needed to preserve the small differences that matter and compress the repetition that does not.

The new path removes obvious noise before extraction, then spends model calls only where the remaining facts show a real duplication problem.

Give every page a history

I rebuilt the crawler around a page ledger.

Each discovered URL now has a record of where it came from, what happened during the crawl, whether its content changed, and which facts came from it. A fact can point back to a specific page instead of only to a whole website source.

That attribution changes the cleanup problem. Chatterbox can remove facts from a page that disappeared without touching facts from pages that stayed. It can skip unchanged pages during an incremental crawl. If a crawl fails halfway through, it can resume from persisted work instead of paying to start again.

The dashboard can also show the business what happened: pages discovered, ignored, skipped as dated, fetched, extracted, or failed. A smaller knowledge base is easier to trust when the owner can see why it became smaller.

Filter before paying for extraction

The cheapest irrelevant fact is the one the system never sends to a model.

The crawler now discovers from sitemaps first and lets the business exclude paths with ordinary glob patterns such as /blog/*, /locations/*, or *.pdf. I chose globs over regular expressions because the control is meant for business owners and agency staff, not crawler engineers.

I tested those controls against my own site. A full crawl discovered seventeen writing pages. Adding /writings* to the ignore list skipped all seventeen while leaving the rest of the site available. An owner can make that content decision without changing the website or asking me to add a special case.

The system warns when an ignore rule removes most of a site and refuses a configuration that removes everything. A typo should not produce an empty crawl that looks successful.

Dated and blog-like pages are skipped by default. The owner can opt in when those pages contain knowledge that should remain searchable. Where a date is available, it stays attached to the page and its facts so the interface can make freshness visible.

This is a product default, not a claim that blogs are useless. A restaurant's current festival menu may matter. A clinic's post announcing hours from three years ago probably should not answer today's visitor. The business can include dated material when it has a reason to maintain it.

Treat URL families as evidence

Location pages usually announce their structure in the URL: /locations/chandigarh, /locations/mohali, /locations/panchkula.

The new pipeline groups those pages into a structural family before considering an expensive semantic merge. That gives Chatterbox a useful clue for free. Facts repeated across the family are likely template material; facts found on only one member deserve more care.

The merge stage runs only when the crawl shows signs of a real fact blow-up. It clusters near-duplicate facts, asks a smaller model to consolidate bounded groups, and protects manual or user-edited facts from automated changes. Facts removed by a merge are archived with a pointer to the surviving fact rather than silently destroyed.

This keeps the costly stage proportional to the problem. A normal site does not pay for a cleanup pass it does not need. A heavily templated site gets structural grouping before model calls begin.

One site became about 75% quieter

On the business website that exposed the problem, the rebuilt pipeline reduced knowledge-base noise by about 75%.

Here, noise means generated question-and-answer pairs that were filtered or archived because they repeated another fact or came from dated content that no longer belonged in retrieval. The 75% figure is the share of the site's original 5,400-plus pairs removed from the searchable corpus. This result comes from one real site; it is not a benchmark for every website.

The facts that remained were easier to trace to their pages and less likely to compete with several copies of themselves during retrieval. Chatterbox also tracks an internal answering-confidence score built from four checks:

1. retrieval confidence;
2. evidence sufficiency, judged by a separate reasoning model;
3. faithfulness of the answer to the retrieved evidence; and
4. generation consistency, measured by analysing chat transcripts after personally identifiable information (PII) is removed in the pipeline.

During a seven-day measurement window ending July 22, 2026, the score for this business rose from 59 before the new ingestion path to 90 after it. Across all 25 businesses in the rollout, the mean rose from 58 to 93. One of those businesses receives variants of “Will you charge me extra money if you visit me in XYZ city?” about 300 times a week, so a more reliable retrieval and generation path matters in ordinary customer conversations.

This composite is an operational signal, not an accuracy percentage or an independent evaluation. The before-and-after comparison is observational and does not by itself establish that Ingestion 2.0 caused the increase. A controlled evaluation against the same question set is the next step. I also want to track how often merged facts would have competed in the top results and whether businesses restore dated sections after seeing what was skipped.

Building with AI: Separate agents mapped the old crawler, reviewed the design, implemented the backend stages, and built the dashboard controls. I set the product boundaries: dated content stays opt-in, glob rules stay understandable, manual facts stay protected, and merging runs only when the crawl shows a blow-up. The 5,400 noisy question-and-answer pairs from a real website decided what the system had to handle.

Small-business websites will continue to contain duplicated pages, old announcements, inconsistent sitemaps, and useful facts buried among all three. Chatterbox has to distinguish those shapes before it can give a visitor a trustworthy answer. I want the crawler to keep the useful business facts, show where they came from, and leave repeated or expired material out of the way.