AI chatbot for business on your own data: how RAG works, when it invents, and how to check

“We want a chatbot” is the most common opening line we hear in AI meetings, and the least useful place to start. An AI chatbot for business running on your own data is not a product you buy. It is a system standing on three things a company either has or does not: the documents, a set of test questions, and a decision about where your data will sit.

This piece does not settle whether you need a chatbot or an agent — that is in “AI agent vs chatbot” — and it gives no prices. It answers the question that comes up just before the contract is signed: where does such a bot get its answers, why does it sometimes invent them, what you need in place, and how to check it before the first customer sees it. External sources were read as at 6 September 2026.

Three things sold as an “AI chatbot for business”

Three different systems go under one name. The first is a rule-based bot: a decision tree with ready-made answers — predictable, it never invents anything, but it will not answer anything outside the tree, and customers ask in ways the script author did not foresee. The second is a language model with none of your company’s data: it converses fluently, answers from public knowledge, and will invent your returns policy without hesitation, because it has never seen it.

The third is a language model with search over your documents — the pattern known as RAG (Retrieval-Augmented Generation; the name was coined by Lewis and co-authors in a paper from May 2020). This piece is about that pattern only, and it is that pattern behind the phrase “chatbot on your own data”.

One distinction: a chatbot in the strict sense is a system whose only output is an answer. It can read from your systems and still be a chatbot; the moment it has to write to them it becomes an agent — a different project, covered in AI agent vs chatbot.

The argument of this piece: RAG does not make the model “know” more. It moves the problem from “does the model know” to two others: “do the documents know” and “will the search find them”.

An AI assistant for staff or a chatbot for customers — not the same project

The mechanics are identical, the risk is not. An internal AI assistant answers staff questions about procedures and instructions: the user can spot a wrong answer, a human is in the loop by construction, and procedures usually contain no personal data. It is the natural first project — an internal knowledge assistant is one of the processes collected in examples of AI automation.

A customer-facing chatbot is the same engine somewhere else: every mistake is public, the content of the conversation almost always contains personal data, and since 2 August 2026 there is the disclosure duty under Article 50 of the AI Act, covered below.

So our sequence is fixed: quality proved internally first, or in “answer for approval” mode where the bot drafts and the consultant sends; only then a conversation with a customer, and only on topics the test set has confirmed. A chatbot in customer service is the last step, not the first — we set out why on our page on customer service automation.

Where a chatbot gets its answers: RAG in four steps

The whole mechanism fits into four steps, and each has one place where it can go wrong.

1. Indexing. Documents are split into chunks of a size the vendor decides. Each chunk becomes a numerical representation (an embedding) and goes into a search index, known as a vector database. Nothing is trained here: the model does not “learn” your documents; the index is a chopped-up copy of them.
2. Search. The question becomes the same kind of representation, and the chunks closest to it are pulled from the index — usually a handful.
3. Generation. The model receives the question, the retrieved chunks and instructions along the lines of “answer only from these chunks”, and writes an answer.
4. Answer with a source. The user sees the answer together with the chunk it came from.

Four places to go wrong: at indexing, the price-list table is cut in half and the headers land in a different chunk from the values; at search, the 2026 price list is joined by an equally “close” price list from 2024; at generation, the model splices two chunks into a sentence that appears in neither; and with no source on the answer, nobody can check any of it.

One point specific to Polish. Search quality over Polish documents depends on how the embedding model copes with inflection and with text typed without Polish diacritics: a question about “faktura korygujaca” has to land on a chunk about “korekty faktur” — the same words in a different grammatical case, and typed without their accents. We give no figures, because they depend on the model and the documents; the conclusion does not — the test questions have to be written by real people, abbreviations and typos included, not by the vendor.

When a chatbot invents — and why RAG does not eliminate it

A language model always answers something; silence is not built into it. It invents — the phenomenon is called hallucination — when the search returned nothing relevant, when two documents contradict each other, when the question is about something the documents do not cover, and when the answer needs live data the index by its nature does not hold: shipment status, stock levels, today’s price.

Two public cases. In April 2025 the support bot of the software firm Cursor, introducing itself as “Sam”, told users that being logged out of additional devices followed from a one-active-session-per-device policy. No such policy existed, and users believed Sam was human (account in Fortune). Co-founder Michael Truell replied publicly: “We have no such policy. You’re of course free to use Cursor on multiple machines.” The second case is about live data: in Moffatt v. Air Canada (2024 BCCRT 149) the airline was liable for what its chatbot told a passenger — the tribunal awarded CAD 650.88 in damages alone.

RAG reduces that risk; it does not eliminate it. In a study from May 2024 of RAG-based legal tools, a Stanford team found wrong answers in 17–33 per cent of cases — better than a general model with no search, but a long way from vendor claims of “no hallucination”. That is a figure from one domain; it shows the direction, not the level your bot will hit.

Safeguards you can check in a proposal:

• “I do not know” permitted and rewarded — a bot penalised for not answering will learn to make answers up.
• Source mandatory — an answer with no chunk behind it is not shown.
• Confidence threshold — below it, a consultant takes the conversation over.
• Scope limited to the indexed documents — the bot declines questions outside it.
• Live data only from the source system — status, price and availability are read at the moment of the question, never from a document.

The rule worth demanding of every vendor fits in one sentence: a chatbot may only assert what it can point to.

Before you start: which documents suit a knowledge base and which ruin it

Before anyone picks a tool, go through the documents that would go into the index. A good document meets five conditions:

• It is current — it has an owner, a date and a version; otherwise the index cannot tell the price list in force from the previous one.
• It is unambiguous — two procedures that contradict each other do not produce no answer, they produce a confident blend of both.
• It holds no personal data. Procedures and terms and conditions — yes. Ticket archives, email and chat logs — mostly no: they are informal, contradictory and full of names.
• It can be read by machine. PDFs with a text layer, text documents, wikis — yes. Scans with no OCR, slide decks, spreadsheets with merged cells — poorly.
• It has no duplicates or parallel versions.

Here is what proposals avoid saying: a “chatbot” project is in practice a “knowledge base” project. Documents that have been put in order stay with the company whether or not the bot survives the pilot — and they are the first real result of a chatbot rollout.

An illustrative example — the figures are made up for this piece, not client data. A trading company, 40 people, 4 in customer service, roughly 600 email enquiries a month. The “Procedures” folder holds 312 files: 96 are duplicates or successive versions of the same document, 41 are out of date — including a price list from 2024 and a returns policy replaced in 2025. That leaves 175 unique files: 60 are fit for the index unchanged, 70 need merging, and 45 should not go into the index at all. The inventory took two working days and was the project’s first deliverable.

The test question set: measuring accuracy before anyone signs

This is the piece missing from proposals, and it is what decides whether a chatbot on company documents works on your data rather than only on the vendor’s demo. Without a test set of your own, you will first judge quality on your customers.

Build a set of 50–100 questions from real tickets, in the proportions the topics occur in the inbox — the same distribution our customer service automation page tells you to count. For each question write down the expected answer and the document that supports it. Add 10–15 per cent of questions with no answer in the documents — those check whether the bot can decline — and a few containing personal data, to check it neither repeats nor stores it.

Score every answer in one of four categories:

• correct with a source — the only one that counts as a success;
• correct with no source, or the wrong source — the content is right, but nobody can check it;
• “I do not know” — right when the documents held no answer, wrong when they did;
• confidently wrong — the only category that costs you trust.

The threshold for that last category is set by the process owner before the pilot, not by the vendor after it. Run the set weekly during the pilot and after every change to the documents: a bot that passed in March and got a new price list in June is a new bot.

The example continued. Sixty questions, ten per topic: order status, returns and complaints, invoices and payments, delivery, product, account. First run against the unsorted folder: 41 correct with a source (68 per cent), 7 correct with no usable source, 8 “I do not know” (5 right, 3 wrong), 4 confidently wrong (7 per cent). After merging 70 documents and removing 45: 49, 4, 6 (all right) and 1 — still not zero. The threshold the owner set: confidently wrong no higher than 2 per cent across two consecutive weekly runs, before any customer sees an answer the consultant has not approved.

Once it is live, two measures remain, the same as for an internal knowledge assistant: time to reach an answer, and the share of answers with a confirmed source. The pilot itself starts in “answer for approval” mode, and loosening the oversight is a decision made on pilot data, not an assumption in the proposal.

Where company data sits during a conversation, and what about GDPR

During a single conversation your data is in three places, and each has to be asked about separately.

The index. A copy of chunks of your documents, sliced up and hosted somewhere. Ask where, in which region, and who has access. If documents containing personal data went in, the index is a set of personal data that nobody has called one.

The model provider. Every question sends it the text of that question and the retrieved chunks. The provider’s role is settled separately for each operation — what that looks like and what to ask of every provider, not only us, is on our page on security and GDPR.

The logs. Full transcripts of conversations. On a customer-facing chatbot these are often the largest set of personal data in the project, because they record whatever the customer typed — including health or financial details nobody asked for.

Hence the design rules. Send chunks, not documents; a procedural answer does not need the customer’s identity, so personal data stays in your systems. Removing a person or a document from the index and from the logs is a capability you have to design, not a side effect of deleting a record in the source system — we went into that in GDPR and LLM implementation. A European region does not by itself close the question of transfers, and a tool is not GDPR-compliant in itself — it is your use of it that is compliant or not.

This describes the technical and organisational side, not legal advice — the legal assessment belongs to your lawyer or data protection officer.

The machine has to introduce itself: Article 50 of the AI Act from 2 August 2026

Someone talking to a bot has to know they are talking to an AI system. Since 2 August 2026 Article 50 of the AI Act requires it, and it covers an ordinary chatbot on a website, not only critical systems (as at 6 September 2026). The provision itself allows an exemption where the machine is obvious from the circumstances and the context of use; we do not use that exemption. A handover to a human, available throughout the conversation, is likewise how we work rather than a requirement of this provision.

Back to Cursor: after the incident the co-founder said AI answers in email support would be clearly labelled from then on. The label costs little; its absence cost more. What else applies from August 2026 is on our page on AI Act compliance. In Poland market surveillance sits with KRiBSI, the national AI supervisory commission, under the Act of 3 July 2026 on artificial intelligence systems (the Polish AI Systems Act; Journal of Laws 2026, item 1003), whose chapter on penalties applies from 28 October 2026.

Help page, widget, in-house assistant or agent: a decision table

Before you compare proposals for a “business chatbot”, compare four routes — the first needs no bot at all.

The first column is not a joke: a help page never invents anything, and on fixed answers it wins on cost. The third is the in-house AI assistant this piece is about. The last is a different project — with permissions, an action log and an approval mode; what changes in it is on our page on AI agent rollouts. If the channel is the telephone, both the arithmetic and the risk look different — we set that out in how much a voicebot costs.

When a chatbot on your own data is the wrong choice

• The knowledge exists only in people’s heads. You are buying a documentation project, and it is worth calling it that.
• The volume is small. A few dozen questions a month will not repay maintaining a knowledge base; if handling them takes two hours a month, it is almost certainly not worth it.
• The answers depend on live data that cannot be integrated. A bot that cannot read status or price at source will guess them from documents.
• The request really means “change something”. That is an agent — priced and supervised differently.
• A help page is enough. See the first column of the table.
• Nobody will own the knowledge base after the rollout. A bot with no owner degrades quietly: the price list changes, accuracy drops, nobody notices for a quarter.
• The goal is that the customer should not notice they are talking to a machine. We do not build that.
• The domain is legal, medical or financial advice, where a confidently wrong answer has consequences nobody in the company will put their name to. We do not automate decisions nobody is accountable for.

We give no prices, because without knowing the process we do not know the scope. What makes up the bill for a chatbot and for an agent is set out in how much an AI agent costs. “Not worth it yet” is a legitimate result of the inventory — and cheaper than a rollout that shows the same thing six months later.

Eight questions for every chatbot vendor

1. Where exactly does the bot get its answers — show the source chunk on each one.
2. What does it do when it does not know?
3. Test it on our sixty questions, not on your demo.
4. Where is the index, where are the logs, in which region, and who has access to them?
5. How will you remove a document or a person from the index and from the logs?
6. How is the machine labelled, and what does the handover to a human look like?
7. Who updates the knowledge base after the rollout, and what happens to accuracy after a price-list change?
8. What changes in the price and in the permissions if the bot has to do something rather than only answer?

What next

The order is always the same: documents, a set of test questions, a decision about data — and only then the tool. If it turns out the bot is not only to answer but to do something in your systems, the questions change — to permissions, an action log and approval; we set those out on our page on AI agent rollouts.

Tell us what questions it would have to answer and where those answers sit today. We will tell you where we would start — including when the answer is: for now a good help page is enough.