Three pages on our site say the same thing: before you start a project, try exporting the last hundred cases of the process as a single table. We say it in the guide on how to implement AI, in the piece on what an AI implementation costs, and on the AI automation page. None of them shows how to run that test for each source separately, or what comes out of it. This piece starts where they stop.
In our implementation methodology, preparing data for AI is part of the first phase — the audit and choice of process — not an item on a pre-start checklist. Below I describe the half of that audit you can do yourselves: five tests for five sources, seven problems those tests can reveal, and a scorecard that ends in a decision. You run the test for one process, not for the whole company. It takes one afternoon per process, and it should be run by the person who runs the process day to day, not by IT — because only that person knows whether the data shows what really happened.
One note on scope: this is about process data — cases, orders, invoices, tickets and decisions. If you are planning an assistant that answers questions from company documents, the criteria are different, and the place to start is whether you need an AI agent (software that carries out a task’s steps itself) or a chatbot that only answers questions; we explain the difference separately: AI agent vs chatbot.
Before you start: a hundred cases, one table and an outcome column
First agree what a single case is: one order, one complaint, one invoice to approve. The test is to gather, for a hundred different cases, everything a person handling them would need into one table, together with how each of them ended.
Take the last hundred closed cases. If the last few weeks were unusual, draw a hundred cases at random from a longer period. If the process is seasonal, sample from the full cycle, for example the last twelve months. When checking sources, also include a few open or abandoned cases, or cases passed to someone else for a decision. They may reveal data gaps that checking closed cases alone could miss. Check the outcome record only for closed cases. If the process has rare but costly cases, also add a few known ones of that kind, because a random hundred may contain none. A hundred cases is a first look, not a measurement: it shows where the data lives and what is missing, but it does not give precise shares. Even with a hundred randomly chosen cases, a share of around one half is known only to within roughly 10 percentage points either way (we give the calculation in the sources). And finding no gaps in a hundred cases does not prove that there are none.
Before the test, answer the three preliminary questions on the scorecard. The first always applies, including when the test stays inside the company: the data controller (the company or person that decides why and how personal data is processed) sets the export scope, identifies the data needed for the test and the data to remove or mask because the test does not need it — personal data, but also, for example, internal prices or margins, including in free-text fields — and makes sure it has been removed or masked. The controller also decides who in the company may use the copy of the data, where it will be stored and processed during the test (for example, only on company systems or the company’s cloud account in an agreed region, not on a personal laptop, a private cloud drive or in a public AI chat), how long it will be kept and when it will be deleted. The second applies only to a file that is to go outside: the controller separately confirms the transfer — who will receive the file, where the data will be processed, who on the recipient’s side will have access, how long it will be kept there and how it will be deleted. As part of that same condition, the company checks whether contracts with business partners and confidentiality obligations allow the file’s contents to be shared with that recipient — this covers partners’ confidential material, not only personal data. The export does not leave the company until both conditions are met. Masking alone does not replace the transfer confirmation, because masked data can still be personal data. I come back to this below, in the third finding.
The last preliminary question concerns material that is not only yours. If the sample includes partners’ material or data from an outside service, first check whether the contract or the terms of use allow you to use it for the test (question 0c). If you don’t know, ask the person responsible for contracts or a lawyer. A “no” means the result is “not yet”: the test can start only once you have confirmed that the company may use this data.
Download the scorecard as a CSV — a plain text file holding a table, with columns separated by commas. It is a blank sheet with the three preliminary questions, the twenty questions from the five tests described below, columns for the result, the findings and the next step, and rows for the final decision. It opens in Excel and Google Sheets. If Excel shows the whole table in one column, import the file through “Data → From Text/CSV” and choose comma as the delimiter.
Part 1. The test, source by source
Each source has four questions. A “yes” means the source passes on that point. A “no” does not disqualify the process. For each “no”, write down what is missing, and assign the number of one of the seven findings in part two if its description fits. If a “no” fits none of them, leave the finding number blank, but describe the gap, its impact on the pilot, the next step (“Next step” column) and who will take it (“Who fixes it” column). The scorecard result remains “not yet” until the gap’s impact on the pilot, the next step and who will take it have all been established. If your process does not use a given source, skip it.
The table layout is not the whole story: a column can exist and still be empty for many cases. For each field needed for the decision, count how many of the hundred cases have a missing value. If each case occupies one row, Excel’s COUNTBLANK function counts the empty cells in that column within the hundred cases. If a case spans several rows, review those rows together and count each case with a missing value once. Also count how many cases have every needed field filled: gaps in different fields can affect different cases, so each field on its own can be within its acceptable level while there are too few complete cases. If each case occupies one row, you can turn on the filter in Excel, untick blanks in each needed column in turn, keeping the earlier filters applied, and count the remaining cases. If a case spans several rows, review all its rows together and count the complete cases. The person running the process sets the acceptable level of missing values and how many complete cases are needed, together with the pilot’s success criterion. In the “What is missing” column, record for each such field the number of cases with a missing value and the acceptable level, and below that list the number of complete cases and how many are needed. If the number of cases with a missing value exceeds its acceptable level, or there are too few complete cases, describe the impact on the pilot, the next step and who will take it. If either threshold has not been set, or any excess or shortage remains unresolved, the result remains “not yet”.
1. Export from the ERP or another system
An ERP is the system in which a company runs orders, stock and invoices. The same applies to a CRM — a system for managing customer relationships — a ticketing system or accounting software: any system in which the process leaves a trace.
- Can a hundred cases be exported to one file without retyping? Passes: a CSV or Excel file in a quarter of an hour. Fails: screenshots, copying one case at a time, or a request to the system vendor.
- Does every row describe the same type of record? It can be a case or an order line. If one case spans several rows, each must carry its case number, and you count a hundred different cases, not a hundred rows.
- Is there a column with the outcome of the case? Approved or rejected, the amount of the correction, who the case was passed to. Without it there is no way to check whether the system’s suggestions are right.
- Are case and customer numbers consistent with those in the other sources? The same order-number format, the same tax ID — not a tax ID in one place and a shortened company name in another. While you are at it, compare the same fields — amount, deadline, status — across sources for a few cases. Different values are finding 2.
2. Mailbox
- Is it clear which threads belong to the process? A dedicated address, folder or label. If orders arrive in five personal inboxes, that alone is a finding.
- Can the data needed for the decision be read from the message, a form or an attachment? An attachment alone is not a fail. It fails if the attachment is an unreadable scan and the body says only “please find attached”.
- Can the thread be assigned to a specific case and linked to its other data? For example, through the order number in the subject line.
- Is the case outcome recorded in the thread or in a linked record in another source? Fails if the decision was given by phone and recorded nowhere.
Even if all four answers are “yes”, check whether email is the only record of the process. If it is, enter finding 7 on the scorecard.
3. Scans and PDF files
- Does the PDF have a text layer? Open the file, press Ctrl+F and search for a word visible on the page. Finding it confirms text at that spot, not throughout the file, so check a few pages. If the search finds nothing, try another word and try copying a sentence into plain text. No readable result means the file will have to be read first. If there is a text layer, compare key values — amount, date, tax ID, order number — with what you see on the page, in at least one file of each document layout: a text layer produced by automatic reading can have wrong digits, and a correct reading of one layout says nothing about the others. Treat mismatches as finding 4, enter the layouts you did not check in the “What is missing” column and record the result as “no”.
- Do you know how many different document layouts appear in the hundred cases? One layout from one sender is a different situation from forty layouts from forty senders.
- Is the file tied to the case? The case number in the file name or folder, not “scan_0042.pdf” in a shared folder.
- Is the data you need legible despite stamps and handwritten notes? A stamp over the amount or a handwritten note where the order number should be is something to know before the project, not during it.
4. Excel on a shared drive
- Does every row describe the same type of record, and every column one piece of information? This is the “tidy data” principle from the data-analysis literature: each variable in a column, each observation in a row. If a deadline or a responsible person is needed to handle the case, each should have its own column; recording them only in “Notes” fails this check.
- Is there a single current version of the file? Fails: “register_final_v2_Anna.xlsx” next to “register_final_v3.xlsx”.
- Is meaning recorded in words as well, not only in colour? If a row’s red colour means “urgent”, that information is lost in a CSV export. You need a column that records it in words.
- Does the sheet have an outcome column? The same question as for the system — and just as important.
5. Notes and calls
- Are agreements from calls written down somewhere? A note in the CRM, a comment on a ticket, an entry in a sheet.
- Does each note carry a date and a case number? Without them it cannot be matched to any row.
- Can the notes be collected and linked to cases? If they are spread across a notebook, a phone and messaging apps, check separately how to collect them from each of those places.
- Does the note say why the decision was made? “5% discount” says what happened. “5% discount, long-standing customer and a late delivery” says which rule applied.
Part 2. Seven findings the test turns up
This is a description of the situations the test is designed to catch — patterns, not accounts of particular companies and not statistics. Every finding follows the same layout: symptom, cause, what to do about it and its impact on the project. The impact takes one of three values:
- blocks the pilot — until it is fixed, the pilot cannot be measured or cannot run at all;
- adds a data-preparation phase — a separate stage of work comes before the pilot;
- adds a rule to the process — something has to be decided and written down, and the solution will apply that rule.
We give the order of magnitude as “days” or “weeks”, without numbers. It is a guide, not a promised deadline: the number depends on your process, and a range given without knowing it would be guesswork.
1. No outcome column
Symptom: the data shows what came in but not how the case ended. There is an order, but no record of whether it was accepted unchanged. There is a complaint, but no record of whether it was upheld.
Cause: the decision is made in a call or an email, and nobody enters its outcome in the table, because until now nobody needed it there.
What to do: first check whether the outcome can be reconstructed from later traces, such as a correcting invoice or a status change in the system. If not, start recording it today, in one column with a few fixed values.
Impact: blocks the pilot until there are recorded outcomes that the person running the process accepts as the reference. We put this finding first because without an outcome you cannot compare the system’s suggestion with what a person did, so you cannot measure whether the pilot worked. Note: a recorded outcome says what was done, not whether it was right. Before the pilot, the person running the process reviews the outcomes in the sample, corrects the ones they consider wrong, and agrees with the supplier on what result will count as a successful pilot. If the rules changed during the period you sampled — for example the price list or the approval rules — note against each case which version applied at the time, and judge the pilot against today’s rules: leave out cases decided under older rules, or correct their outcome to the current rules while keeping a record of the original decision. Replace the cases you leave out with cases decided under the current rules, so that the sample still covers the full cycle of the process and the rare but costly cases. If there are not yet enough such cases, the scorecard result is “not yet” until they accumulate. Who fixes it: you. Order of magnitude: days if the outcomes can be reconstructed; weeks if you have to wait for new ones to accumulate.
2. Three versions of the truth
Symptom: the same case has one amount in the system, another in the spreadsheet and a different deadline again in the email. Each source on its own looks correct.
Cause: data is retyped by hand between places, and a correction reaches only one of them.
What to do: for every field, name the source you trust — for example the amount from the system and the deadline from the email confirmation. This is a decision about data, not about who in the company is right.
Impact: adds a rule to the process. Who fixes it: you decide; the supplier builds the rule into the solution. Order of magnitude: days.
3. Personal data in the “notes” field
Symptom: a free-text column, a note or an email body contains phone numbers, addresses, health details or family circumstances — things the process does not need at all.
Cause: a free-text field accepts anything, and people type whatever is to hand.
What to do: before the export leaves the company, remove or mask unnecessary personal data within free-text fields, keeping the information the test needs. What to mask and how is decided by the data controller — with the data protection officer (DPO), if the company has appointed one. The data controller separately decides whether the export may be sent to the supplier or model, before anything is sent; masking alone does not replace that decision. The solution can include an intermediate layer that removes or masks the identified unnecessary data before it is sent to the model; we describe it in our piece on the GDPR, the General Data Protection Regulation, in the section “Send less”.
Impact: adds a rule to the process. Who fixes it: the decision is yours; the supplier builds the filter. Order of magnitude: days.
4. Scans without a text layer, or with errors in it
Symptom: on the pages checked, neither searching for visible words nor copying sentences gives readable text — the data you need is stored as an image. The documents are photos of pages, sometimes skewed, sometimes with a stamp over the amount. Or there is text, but key values — amount, date, tax ID — do not match what you see on the page.
Cause: documents arrive on paper, or someone scans them to an image rather than to text.
What to do: reading the document becomes a separate step that has to be tested on your files before the rest is built. It is also worth asking senders whether they can send electronic files instead of scans. For invoices there is another route as well, described below.
Impact: adds a data-preparation phase. Who fixes it: the supplier builds and tests the reading, you talk to the senders. Order of magnitude: days for a single document layout, weeks for many.
5. A system with no export
Symptom: there is no way to get the data out except by screenshot or retyping.
Cause: an old system, an edition without a reporting module, or a licence that does not include access to the data.
What to do: check three routes: a report saved as a file, an API (an interface through which other software can fetch data from the system) and an export built to order by the system vendor.
Impact: blocks the pilot until one of those routes works. Who fixes it: you, together with the system vendor. Order of magnitude: days or weeks — depending on which route is available, how much work it needs and how fast the system vendor answers.
6. Mismatched identifiers
Symptom: the email says “order 123”, the system says “ZAM/2026/000123”, and the spreadsheet has the customer’s name instead of their tax ID. Every case can be matched by hand, but not automatically.
Cause: each source was set up separately and nobody agreed how to write the numbers.
What to do: write down the matching rule — which number is the master and how to convert the others — and where no rule can be found, keep a simple mapping table.
Impact: adds a rule to the process. Who fixes it: the supplier, from your list of cases. Order of magnitude: days.
7. The process lives in email
Symptom: the whole case — enquiry, agreements, decision — sits in email threads written as free text. The system shows only the outcome, or nothing at all.
Cause: email is the simplest tool and nobody needed anything more.
What to do: for new cases, introduce a form or a fixed message template with a few fields. The email history can still be used, but reading it is a separate piece of work.
Impact: adds a data-preparation phase. Who fixes it: the form — you; reading the history — the supplier. Order of magnitude: days for the form or for reading the history; weeks if new data has to accumulate.
The seven findings in one table
| Finding | Impact on the project | Who fixes it | Order of magnitude |
|---|---|---|---|
| No outcome column | Blocks the pilot | You | Days or weeks |
| Three versions of the truth | Adds a rule to the process | You decide, the supplier builds it in | Days |
| Personal data in the “notes” field | Adds a rule to the process | You decide, the supplier filters | Days |
| Scans without a text layer, or with errors in it | Adds a data-preparation phase | The supplier and you | Days or weeks |
| A system with no export | Blocks the pilot | You with the system vendor | Days or weeks |
| Mismatched identifiers | Adds a rule to the process | The supplier | Days |
| The process lives in email | Adds a data-preparation phase | You and the supplier | Days; weeks if new data has to accumulate |
Two changes that reduce data preparation
Purchase invoices: KSeF instead of scans
Invoices received through KSeF, the Polish national e-invoicing system, arrive as a file with a fixed structure rather than as an image, so for them reading a scan can be skipped — what that means for bookkeeping we describe under accounting automation.
Orders and requests: a form instead of free-text email
A form with fields for the customer number, the order number and the deadline needs no AI at all, and it brings order to new cases from the day it is introduced. For it to remove mismatched identifiers as well, the form must use the same numbers as the system. The outcome of each case still has to be added once the case closes.
The scorecard: pilot, data preparation or “not yet”
After the test you have the answers to the preliminary questions, the test results and a list of gaps — with or without a finding number. The decision follows from the worst of them:
| Test result | What it means | Next step |
|---|---|---|
| All required checks satisfied, case outcomes recorded, no unresolved gaps or work required before the pilot | The data is ready for a pilot | A pilot checked on set-aside cases (below) |
| The remaining work adds a rule or a data-preparation phase, but no gap blocks the pilot | Data preparation: a pilot is possible, but not straight away | A data-preparation phase covering the listed gaps |
| A preliminary question not answered “yes” — for 0b and 0c, “not applicable” also passes — or at least one gap blocks the pilot, or its impact, next step or who will take it has not been settled, or the acceptable level of missing values in a needed field has not been set | Not yet | Fix or explain it, then rerun the test |
If the result is “pilot”, set some cases aside — including a few rare but costly ones, if you have them — before the supplier starts building, and do not share them with the supplier while the solution is being built or adjusted. Only when the pilot is checked does the solution receive those cases’ data, without the approved answers and only as it stood when the case was decided — without later statuses, corrections or notes from which the outcome could be read. The person running the process compares the solution’s suggestions with the approved answers against the success criterion agreed beforehand.
A “not yet” result needs the obstacle to be explained or removed, and where a recorded decision history is missing, the rule from our what we don’t do page applies: “If the decision history exists only in somebody’s head, it has to be written down first”.
A completed scorecard is also the best material for a first conversation with a supplier: it shows the state of the data in your process, so the conversation can be about scope rather than guesswork. We set out how scope and cost follow from the state of the data in the piece on what an AI implementation costs, and the whole project on the AI implementation page.
Sources — as of 12 October 2026
- NIST/SEMATECH e-Handbook of Statistical Methods, 7.2.4.1 “Confidence intervals” — the Wilson confidence interval for a proportion, at the 95% confidence level. For a hundred randomly chosen cases and a share of around one half, the half-width of the interval is about 9.6 percentage points; with zero gaps found, the upper limit is about 3.7%. Our own calculation from the handbook’s formula; it applies to a random sample, not to an arbitrary set of recent cases.
- Hadley Wickham, “Tidy Data”, Journal of Statistical Software 59(10), 2014 — the principle: each variable is a column, each observation is a row.
- Microsoft — COUNTBLANK function — counting empty cells in a range; read on 12 October 2026.
- Regulation (EU) 2016/679 (GDPR) — Article 5(1)(c) (data minimisation) and Articles 37–39 (data protection officer); read on 12 October 2026.
- Scope of mandatory KSeF — Polish Ministry of Finance page, in Polish: receiving invoices through KSeF has been mandatory since 1 February 2026, subject to statutory exclusions; the transitional rules until the end of 2026 concern issuing invoices, not receiving them; read on 12 October 2026.