Why Chatbots Make Things Up About Your Business

A customer asks your website assistant whether you take dogs. You have never written anything about dogs. The assistant answers anyway, warmly and in your brand voice, and tells them yes.

You find out on a Saturday, when somebody arrives with a labrador and a screenshot.

This is the failure that matters when a business puts AI in front of its customers, and it is not the one most people prepare for. The worry is usually that the assistant will not know enough. The actual risk is that it will answer regardless.

The Problem Is Confidence, Not Ignorance

A general language model is built to produce likely text, not true text. Ask it about a refund window it has never seen and it will not stop. It will give you the most plausible refund window, which is thirty days, because that is what most businesses use.

Fluency reads as knowledge. That is what makes it expensive. Nobody double-checks an answer that arrives in confident, well-punctuated sentences on your own website, under your own logo. A hedged answer gets verified. An invented one gets believed, quoted back to you, and occasionally enforced.

Grounding Is Retrieval, Not A Better Prompt

The usual first fix is to write a sterner instruction. Be accurate. Do not make things up. Only state facts.

It does not work, because the model has no way to check itself against anything. You have asked it to be careful without giving it a thing to be careful about.

The fix is to stop asking the model to know your business and start handing it the answer. When a question arrives, search the business's own content for the passages that bear on it, give those passages to the model, and ask it to write the reply from them and nothing else. The model stops being the source of the facts and becomes what it is good at: turning the right facts into a sentence a person wants to read.

This is what we built Kakapo to do. A business trains it on the content it already has, meaning its website pages, uploaded documents, plain text, and question and answer pairs. Every question then goes through retrieval first, and the answer is written from what comes back.

The Limit Has To Go Last

Two sets of instructions reach the model on every question. One belongs to the business and sets the assistant's voice. The other is the rule that keeps it grounded.

In Kakapo the grounding rule is added after the business's own instructions, deliberately, and this is more important than it sounds. A business writing its own prompt will eventually write something like "you are an expert who always has an answer". Put that last and it quietly cancels the rule that stops the assistant inventing a price.

Which instruction is allowed to win cannot be left to whoever wrote the more emphatic sentence. The assistant is answering as somebody's business, and inventing a refund window is worse than admitting to a gap.

"I Do Not Have That Information" Is A Feature

So that is what Kakapo is told to say. When the retrieved content does not contain the answer, it says it does not have that information and suggests contacting the business directly. It is told never to invent details, prices, dates, or policies.

Read as a demo, that looks like a limitation. Read as a business owner, it is the whole product. A miss costs you a click and a reply. An invented policy costs you the refund, the argument, and the review.

A miss is also worth more than it looks, because it is a list. Every question the assistant could not answer is a question your customers are actually asking and your content does not address. That is the most useful content brief you will ever get, and you cannot have it from an assistant that answers everything.

You Have To Be Able To See What It Said

An assistant whose answers you never read is one you are trusting on the strength of a sales demo.

Every Kakapo answer carries a relevancy score alongside it, and conversations are logged, so an owner can sort by the questions that were handled worst rather than skim the ones that went well. Half an hour in those logs tells you what to write next.

Grounding does make the assistant only as good as your content. That is the point. It turns answer quality into a content problem, which is a problem you can see, own, and fix, rather than a model problem you can only complain about.

Your Content Answers For You, And Only For You

One more thing worth insisting on. Retrieval never crosses from one assistant or one account to another, so your pricing, your policies, and your documents can only ever answer for you. A shared knowledge pool is convenient right up to the first time it is not.

What To Ask Before You Buy One

Three questions separate a grounded assistant from a fluent one, and they work on any vendor.

• Which of my content does it answer from, and what happens when I change it? If the answer is vague, the content is decoration and the model is guessing.
• What does it do when the answer is not there? You want a vendor who can describe the refusal, not one who tells you it always finds something.
• Where do I read what it actually said? If there is no log and no measure of quality, you will learn about the labrador from the customer.

An assistant that answers everything is easy to build and easy to demo. One that knows where your knowledge stops is the only kind worth putting in front of a customer.

If you are weighing one up, our AI workflow integration work starts from the content you already have. Tell us what your customers keep asking and we will tell you honestly whether an assistant is the right answer for it.