What is RAG, and when does it make sense on company data — and when is search or an export enough?

RAG is the acronym that appears most often in offers for an “AI assistant on company documents” and is explained least. This piece explains what it is, without jargon, and — more importantly for the decision — when it makes sense and when an ordinary search, passing a few files in full or exporting data to a spreadsheet is enough. If an assistant is to work on documents or data from your systems, that is exactly the kind of connection we build as part of connecting an AI assistant to your system.

This piece does not repeat how a chatbot on documents works step by step or when it makes things up — we covered that in the piece on an AI chatbot on company documents. Here the question comes earlier: whether to build one at all. We read the vendor documentation cited here on 12 October 2026.

RAG in three sentences

RAG (Retrieval-Augmented Generation) works like this:

1. Search first. The system searches your documents, indexed beforehand (more on that below), and picks a few passages that match the question.
2. Then answer. A language model — the engine behind assistants such as ChatGPT or Claude — gets the question together with those passages and writes an answer based on them.
3. Cite the source. The user sees which document and passage the answer came from, so they can check it. A source makes checking easier, but it does not guarantee that the answer is correct.

Anthropic describes RAG as a method that retrieves relevant information from a knowledge base and appends it to the question, and Microsoft describes it as a pattern that grounds the model’s responses in your own content. Citing the source has to be designed in as a requirement, and model vendors have ready-made features for it — for example, Anthropic describes citations that return passages meant to support each claim.

How this differs from uploading files to ChatGPT

A few short documents can be passed to the model in full if the model can accept that much text in one request. That works, and with a few documents it is the simplest option. Uploading files to a chat does not tell you how the service uses them, though — with larger files it may search passages itself — and manually uploading a copy of a file does not carry over the source’s permissions or keep it updated when the source changes. RAG starts to be needed when there are too many documents to fit into one request: in the most common variant the system then keeps them in a search index — a copy of the documents cut into passages. With search by meaning, each passage gets a numerical description of its meaning (an embedding), so it can be found even when the question uses different words from the document; this is often combined with ordinary keyword search.

The most common misunderstanding: RAG does not “teach the model your company”. The model is not trained on the documents; for each question it gets only a few retrieved passages. Whether the model provider may use submitted content for training is a separate matter, settled by the terms of the specific service, not by RAG itself.

When RAG makes sense

RAG makes sense above all when the first of the conditions below holds; each further one strengthens the case:

• There are many documents. Enough that they do not fit into one request to the model, or that passing them in full with every question would be slow and expensive. Microsoft gives the example of 10,000 pages of documentation that makes no sense to send with every question.
• The documents change. Procedures, instructions, terms and price lists get new versions. Google points out that a model on its own knows only the data it was trained on, and RAG provides it with up-to-date information — provided the index is refreshed together with the documents.
• The answer needs a source. An employee asking about the complaints procedure or a clause in a framework agreement must be able to click and see the document. Without a source there is no way to tell a correct answer from a convincing-sounding one.

Typical examples: a knowledge assistant for the customer service or field service team that answers from procedures and instructions; finding clauses in a large set of contracts; product technical documentation that nobody knows in full.

When RAG does not make sense — and what to do instead

There are a few documents: pass them in full

If all the knowledge fits into a few documents, nothing needs indexing. Anthropic wrote in September 2024 that a knowledge base smaller than 200,000 tokens (about 500 pages) can simply be included in full in the prompt, with no need for RAG. A token is the unit of text in which a provider measures the length of a request and the charge. Google says something similar: a model’s long context window — how much text the model takes in at once — is a good way to provide source material, and RAG is worth adding when there is more material than fits, or when cost and response time need to come down (Google Cloud). Limits depend on the model and change, so check them in the chosen model’s documentation on the day of the decision.

The questions are about numbers in a system: query the system or export

“How many unpaid invoices do we have from this customer?”, “what is the stock level of this item?” — these are not questions for documents but for data in tables. RAG looks for passages of text, so it answers such questions poorly: it may land on a passage of a month-old export instead of counting. There are two better routes:

• A plain export. If the question comes up once a week, an export to a spreadsheet or a ready report in the system is usually enough. Before building anything, compare its cost and accuracy with what an assistant would cost and how accurate it would be.
• A query to the system at the moment of asking. When the data must be current to the minute, the assistant should read it from the system — the ERP (business management system), the CRM (customer and sales system) or the accounting software — through its interface (API, the way programs exchange data) or through an MCP server, a small program that gives the assistant selected data and functions of the system within the scope it has been allowed. We describe four ways of making that connection in the piece on AI integration with ERP and CRM.

In practice a good assistant combines both worlds: it searches documents for procedures and instructions, and reads current statuses, prices and stock levels from the system at the moment of asking, not from a document.

Search already works: leave it

If employees type the name of a procedure into the intranet or drive search and get the right file, and a link to the document is all they need, a RAG assistant adds little. An assistant may be useful, for example, when the answer has to be assembled from several documents or people do not know what the document is called. An honest test: ask a few people how they look for answers today and how long it takes. If the answer is “I type it into search and there it is”, this is not a RAG project.

Permissions first

A search index is a copy of document passages. On its own it does not know who may read which document — if permissions are not carried over into it, the assistant may quote an employee a passage from a file they should never have seen.

Vendors say so plainly. Once it is set up to copy permissions into the index and filter results, Azure AI Search captures permission information during indexing and enforces it at query time — provided the application passes on who is asking. With SharePoint, how permissions are updated in the index depends on the integration version and the kind of change; some changes, such as permissions granted on a whole site, library or folder, need a forced refresh, otherwise the index shows documents under the old permissions. And Microsoft 365 Copilot surfaces to a user only the data they have at least view permission for — which also means that the assistant may show a file accidentally shared with the whole company.

Two rules follow from this, and we start every such project with them: the assistant searches only what the person asking could open themselves, and before anything goes into the index, someone checks who has access today to the folders that are to be searched.

A quality test on your own documents

Whether RAG works is decided not by a vendor demo but by a test on your documents and your questions. For a first check of an internal assistant, 30–50 real questions are enough:

1. Collect questions from the people who ask them — with abbreviations and typos, in the proportions in which topics come up at work.
2. For each question, write down the reference answer and the document passage that confirms it. Also record who is asking and which documents they can access — the same answer may be available to one person and not to another.
3. Add a few questions with no answer in the documents, to check whether the assistant can say “I don’t know” instead of guessing.
4. Measure three things separately: the share of questions with an answer available to the person asking for which the search found the supporting passage (the hit), whether the answer is correct and whether it cites the right source. Assess refusals on questions with no answer separately, and check separately that answers and citations do not reveal content beyond the asker’s permissions. These measures help you tell whether the problem lies more in the documents and the search or in the answer itself.

The threshold the assistant must pass is set by the person responsible for the process before the test, not by the vendor after it. The set is rerun after every larger change to the documents. For a customer-facing chatbot the set should be larger, with four answer categories — we set that out in the piece on an AI chatbot on company documents.

What the cost is made of

We will not give an amount, because it depends on your documents, but you can estimate it yourself from five lines. This is a simplified estimate: the token prices you plug into the formulas are quoted per million tokens, and which charges apply depends on the services you choose.

An example rate from a price list: OpenAI bills storage of indexes in its search service by size — the first gigabyte, counted across all indexes, is free, and every further one costs USD 0.10 per day (OpenAI documentation, as of 12 October 2026). Other providers charge differently, so the lines are compared in the price lists of the chosen services, not carried over from someone else’s offer.

The last line of the table is often left out of offers, yet it decides whether the assistant still answers in line with current procedures six months later. The index does not know that the price list changed until someone replaces the document.

Data and GDPR: where the index lives and for how long

With RAG, company data may sit in four places: in the index (a copy of document passages); in the service that creates the embeddings (the numerical descriptions of meaning) — if it runs at an external provider, it receives every indexed passage and, with search by meaning, every question too, even when the index and the answering model run on your side (Microsoft describes using the same model to create embeddings at indexing and at query time); in the model’s environment — your own, or the provider’s, which receives the question and the retrieved passages with every question; and in the conversation logs, if they are kept. GDPR, the EU General Data Protection Regulation, applies to each of them if personal data is there. Each needs its own questions: which region it is in, who has access to it and how long it is kept.

The point most often overlooked is what happens to the index after a document is deleted. When a document is deleted from the source, it must also disappear from the index — otherwise the assistant keeps quoting it. Syncing deletions has to be ensured and tested: configure a ready-made mechanism if the service has one, or build your own. Some ready-made mechanisms are broader: for example, OpenAI lets you set the whole index to expire after a period without use, but that does not replace removing individual documents deleted at the source. If documents containing personal data are to go into the index, the company decides on it after an assessment by its lawyer or data protection officer. What follows from this when deploying language models we describe in the piece on GDPR and LLM implementation, and how we treat data in our projects on the security and GDPR page.

When not to build, and where to start

We would not build RAG when:

• there are a few documents and they fit in full into one request to the chosen model — passing them to the model in full is enough;
• the questions are about data in a system — a report, an export or a query to the system is better;
• search already works and a link to the document is all people need;
• nobody is responsible for keeping the documents current — the assistant will confidently quote outdated versions;
• folder permissions are disorganised — sort out access first, then build the index.

If none of these points fits, we start with a narrowly scoped pilot: one set of documents, one team, answers checked by a person. The assistant suggests, the employee decides, and the quality test built from real questions shows whether it is worth going further. How we run such a project from choosing the process to handover we describe on the page on AI implementation for business.

Tell us which documents or which system the assistant should work on. We will tell you whether it is a RAG project, a query to the system or a job for ordinary search — including when the answer is “not worth it yet”.