Why Chatterbox Refuses to Guess

Suppose a visitor asks whether a shop delivers to their postcode. The website mentions delivery but never lists the service area. A general chatbot can turn that weak clue into a confident “yes.”

The owner discovers the damage after the visitor has ordered, travelled, or left. The same failure can invent a price, promise an appointment time, or describe a return policy the business never offered.

Chatterbox uses a narrower contract: answer from the business’s supplied content or admit that the answer is missing.

Retrieval comes before the reply

The business adds FAQs, website pages, and documents to its knowledge base. Chatterbox breaks the material into searchable chunks and creates embeddings for retrieval.

When a visitor asks a question, the system first searches that tenant’s content. It gives the answer model the relevant pieces and instructs it to treat them as the source of truth.

The model does not receive another business’s knowledge. Tenant scoping applies to ingestion, retrieval, conversations, and leads.

Two checks protect the answer

Finding a vaguely related paragraph is not enough.

Chatterbox checks whether retrieval found useful material, then asks the answer stage to decide whether that material supports a response. A weak result follows the refusal path.

This handles two common failures:

• retrieval found nothing relevant;
• retrieval found text, but the text does not answer the visitor’s question.

The second case matters. A page mentioning “delivery” does not prove that the business delivers to the visitor’s postcode.

Refusal should stay useful

An honest refusal can still help the visitor move forward.

Chatterbox can say that it does not have enough information and offer a contact form. The visitor leaves a phone number or email. The resulting lead includes the conversation and the page where the question started, so the owner can follow up with context.

The widget closes the captured conversation after a successful submission. The visitor can start a new thread if they have another question. This keeps the completed lead separate from later chat and makes the handoff state clear.

The product does not claim that a live agent has joined. A small business may have one owner answering leads between other work. Contact capture is a more honest bridge.

Refusals reveal missing content

A repeated refusal is product information for the business.

Repeated refusals are useful decision-level evidence: the request completed, but the product chose not to answer.

If visitors keep asking about installation, service areas, bulk pricing, or a return policy, the dashboard can show that gap. The owner can add an FAQ or update the website, then reprocess the knowledge base.

This creates a maintenance loop:

1. a visitor asks a question;
2. Chatterbox cannot support an answer;
3. the business sees the missing topic;
4. the business adds the correct information;
5. later visitors receive a grounded answer.

The refusal becomes a way to improve the website instead of a dead end.

Freshness matters as much as retrieval

Grounding cannot protect a business from its own stale content. An old price in the knowledge base can produce a well-grounded, wrong-for-today answer.

Chatterbox therefore needs crawl status, source management, and freshness reminders beside the chat system. Owners and agencies should be able to see where an answer came from, update the source, and re-ingest it without filing a support request.

Agency accounts make this operational problem larger. A web shop managing several client sites needs low-maintenance knowledge updates or the support cost consumes the reseller margin.

Multilingual answers keep the same boundary

A visitor may ask in English, Hindi, or Hinglish. Chatterbox can normalize the question for retrieval and answer in the visitor’s language.

Answering in another language should preserve the same evidence requirement.

Translation should not turn an unsupported answer into a supported one. Per-language evaluation is needed because good English retrieval can hide poor performance in another language.

The boundary is part of the product

Many chatbot demos optimize for how often the bot responds. A small-business product has to care whether the response is safe to act on.

Chatterbox’s refusal contract accepts a shorter conversation when the knowledge is weak. The visitor gets a clear path to the owner. The business learns what information is missing. The bot avoids borrowing confidence from a general-purpose model.

The useful answer rate may be lower. The business gets a record of what visitors still need, and the visitor gets a path to a person instead of a plausible invention.

Building with AI: I chose the trust boundary and used AI to implement much of retrieval, prompting, and ingestion. The answer model's default instinct is to be helpful, so a prompt instruction alone is too easy to weaken during later edits. The deterministic similarity floor, no-model-call refusal, INSUFFICIENT_INFO contract, and regression tests keep that product decision outside the model's discretion.

The refusal path in code

The boundary is enforced at two separate points in apps/rag/orchestrator.py.

First, retriever.retrieve_chunks performs tenant-scoped vector search and applies RAG_MIN_SIM. If no fact clears the floor, answer_question returns the refusal without calling the answer model. This saves a provider call as well as avoiding an unsupported answer.

Second, the answer model receives only the retrieved fact block plus the bounded conversation history. An empty response or the literal token INSUFFICIENT_INFO takes the same refusal path. Tooling can therefore distinguish “nothing relevant was retrieved” from “evidence was retrieved but did not support an answer” without showing that internal distinction to the visitor.

The ingestion side protects the evidence before it reaches retrieval. Facts are deduplicated per tenant using normalized text and pair embeddings. Manual and user_edited facts cannot be replaced by an automated crawl. Recrawl cleanup can delete vanished auto-detected facts, but it is disabled for cancelled or empty extractions.

I exercised tenant isolation, no-hit refusal, insufficient-answer refusal, answer-aware deduplication, manual-fact protection, conversation-history limits, and stale-fact cleanup. An unsupported question is now a predictable state with a lead handoff rather than an invitation for the model to improvise.

The similarity floor and INSUFFICIENT_INFO contract sit outside the prompt. A later copy edit cannot silently remove either boundary.