An AI agent is a system that not only answers but also carries out actions in a company’s systems itself — it looks up documents, records data, prepares messages. Its pilot usually starts with “let’s run it for a few weeks and see how it goes” and ends with “it looks promising”. No decision can be based on that, because nobody agreed in advance what would settle it. This article describes a pilot as a protocol written down before the first day: two stages on real cases, a case count that follows from a threshold, four numbers, stop rules and a one-page decision for each category of the agent’s actions. At the end there is a filled-in example of an agent that assembles the documents for a complaint.
Three things are not repeated here. Where the person stands in the process and why you start by approving every case is step four of implementing AI; why a new system first runs alongside the existing work is step six of the same article. How to design the approval screen itself, how to tell it has become rubber-stamping, and when to loosen oversight of a category once the system is live is the article on human approval. Here we deal with what happens in between: how a pilot collects its numbers and how the decision “go live or stop” follows from them. Where the pilot sits in the whole project, and that it can end with “not worth it”, is described on our methodology page.
First, split the agent’s actions into categories
Because an agent acts in systems — how it differs from a chatbot is set out in our comparison of an AI agent and a chatbot — an agent pilot does not assess “the agent” as a whole. It assesses each category of actions separately, because each carries a different risk, and an averaged result hides the category that gets things wrong. In practice three are enough:
| Category of actions | Example | Can it move to approving only exceptions |
|---|---|---|
| Read | Finding the order and invoice for a complaint | Reading on its own changes no data, so it does not wait for approval; you measure whether the agent found the right document |
| Internal, reversible write | A note in the complaint record listing the missing documents | Yes, but only after go-live, on data from day-to-day work — and still under oversight |
| External or irreversible action | A message to a customer, a price, a booking, a commitment, a payment, deleting data | Not in the first version, whatever the results; irreversible actions — sending to a customer, a payment, a deletion — never |
The last row is a principle, not a pilot result: in the first version, actions that leave the company or cannot be undone stay with a person even if the agent never got one wrong. We do not hand the irreversible ones to the agent in later versions either. We describe the same split into read, write and operations that need a person’s approval in what an AI agent is, and on the AI agents service page as the first decision in a project.
Two pilot stages and what each measures
A pilot has two stages, and the third that is often added to it is in fact its outcome.
Stage 1: shadow mode. The agent receives the same cases as the staff and reads the same systems, but every writing action is only recorded: the log shows what the agent intended to do and with which data, and nothing changes in the system. People work as before. After each case the agent’s intention is compared with what the person did. In this stage the agent changes no data in the systems and sends no messages — but it still reads real customer data, so access to that data and where it goes have to be controlled just as they will be later. From the first day the stage gives agreement with the person, the exception rate and the cost of one case. One thing has to be checked before the start: that the interception of writes works independently of the model. An instruction to the model saying “do not write anything” is not a safeguard but a request. For the same reason, before the first real case the agent needs a list of permitted tools and of the places it may send data to, checked outside the model — shadow mode protects against writes, not against reading data and sending it where it should not go.
Stage 2: the agent proposes, a person approves. Every writing action goes into a queue and runs only after approval. The employee no longer does the case from scratch but checks a proposal. A fourth number appears: how often the approver corrects or rejects something. This stage also shows a cost that shadow mode cannot see — people’s time spent approving.
Stage 3 is not part of the pilot. Moving a category from approving every case to approving exceptions happens later, on data from day-to-day work and against the thresholds described in the article on human approval. At most, a pilot can name the internal, reversible categories that are candidates for that move. A hundred pilot cases are too few to remove oversight from them, and approval during a pilot is usually more careful than in the third month of everyday work.
How many cases are enough: the number follows from a threshold you set in advance
“How many cases” has no single answer, because it depends on what you want to show. Start with a sentence such as “in this category the agent is wrong in fewer than 5 cases in 100” — and you set that value before the start, because it is a decision about how many mistakes the process can tolerate, not a technical parameter. Before the start you also agree which cases go into the assessment — all of them in the period, for a given version of the agent — and how many there should be or when the assessment ends. The result is checked once, at the end of that period. Looking every week and ending the assessment in the first week in which the bound happens to fall below the threshold overstates the confidence of the result. The number of cases then follows from arithmetic.
Throughout this article we use one method for the upper bound — the Wilson interval, which the NIST statistics handbook recommends for proportions from a sample, calculated for a two-sided 95% interval. When the whole assessed sample had no error at all, it gives a simple answer to how many cases you need to assess:
| Threshold you want to show | Assessed cases, if the whole sample had no error (approx.) |
|---|---|
| below 10 in 100 | 35 |
| below 5 in 100 | 73 |
| below 2 in 100 | 189 |
| below 1 in 100 | 381 |
You will often meet the gentler “rule of three” (Hanley and Lippman-Hand, JAMA 1983): the upper bound is roughly 3 divided by the number of cases, so 30, 60, 150 and 300 cases for the same thresholds. It gives smaller numbers because it corresponds to a one-sided bound. Both are honest; what matters is not to mix methods within one decision — in this article we stick to Wilson.
This table does not let you wait for a “good run” and drop earlier errors. An error stays in the results for that version of the agent; counting starts again only after the agent changes, as described below.
Once errors have appeared, the result can be sobering: with two errors in 100 cases the observed rate is 2%, but the upper bound of the 95% interval is about 7%. That is why, at the go-live decision, the error rate is compared with the threshold through its upper bound, not the raw result.
Three practical rules follow:
- Count each category and each stage separately. A hundred cases in a pilot are a hundred cases in a category that occurs in every case, but may be only twenty in a category that appears in one case in five. For the decision about day-to-day work, only stage-2 cases count, and only those in which the agent made a proposal — exceptions reduce that number further.
- Volume limits what a pilot can show. A category with 40 cases a week gives 160 cases in four weeks, but if two of those weeks are shadow mode, about 70 proposals remain for the decision about day-to-day work — at best a threshold of about 5 in 100. A category with five cases a week gives ten cases in two weeks of stage 2 — even without a single error the upper bound is then about 28 in 100. That is why a few dozen cases may be enough to assess an action whose mistake someone will correct at the next step anyway, and not enough for an action whose mistake costs money — there the threshold is stricter, and a pilot usually does not gather that many cases.
- Every case in the period counts, not selected ones. The arithmetic assumes the cases are comparable and are not picked by someone who knows which are easy. When errors come in runs — for example, all of them concern one supplier — the result is too optimistic.
The pilot’s four numbers
All four numbers are tracked separately for each category, every week. The upper confidence bound is compared with the go-live threshold once, at the end of the agreed assessment period. Weekly exception thresholds, per-case cost ceilings and the other stop rules are checked throughout the pilot. Below we describe how to count them; we agree their thresholds with a client before the start, choose them for the process, and they are not market or benchmark norms.
- Agreement with the person (stage 1). The share of cases in which the action the agent intended is the same as what the employee did. “The same” has to be defined before the start — for documents, usually field by field. The person is not always right, so every disagreement is reviewed by a third person and put in one of three groups: the agent is wrong, the person is wrong, or both results are acceptable. Only the first group counts as an agent error, and it is the agent’s error rate that is compared with the threshold. Agreement does not prove correctness — the agent and the employee can make the same mistake — so the third person also reviews a random sample of agreed cases. An error found in that sample is an agent error even if the employee made the same mistake. Do not scale it up to the whole group or count it once: if the audit finds even one, check all agreed cases before the decision and only then count the agent’s errors. The audit result and the sample size are reported separately.
- Correction and rejection rate (stage 2). The share of proposals the approver corrected or rejected. We describe the same number in the article on human approval; that article also explains why on its own it says little and has to be read together with the time to decide and with an audit of a random sample of approved cases — separately for each category — done by someone other than the approver. Without that audit, zero corrections may mean nobody was reading. For the pilot, separate substantive corrections — a wrong document, a missing requirement — from cosmetic ones such as word order.
- Exception rate (both stages). The share of cases the agent did not finish and handed to a person without a proposal: data was missing, it exceeded the limit on the number of actions it may take in one case, or it decided it was not sure. Too high means the agent takes nothing over. Suspiciously low is also a signal — check whether the agent hands a case over when it should, for example when a complaint has no invoice number. The agent makes no proposal in an exception, so agreement and the correction and rejection rate are counted only over cases with a proposal, while the exception rate and cost are counted over all cases, including the time spent handling exceptions.
- Cost of one case (both stages). It has two parts, best shown separately, because money and minutes cannot be added without an hourly rate. The first is the fees for using the AI model, including loops and retries — an agent that searches for the same order ten times pays for every repetition. The second is people’s average time per case: approval plus handling exceptions. You compare both with the cost of the same case before the pilot, measured at the very start (that is step two of implementing AI). How to break down model and oversight costs is shown in our article on what an AI agent costs. This cost of one case settles whether the agent works in practice; whether the whole implementation pays off is settled only by a calculation that adds the fixed costs — platform, monitoring and upkeep — set out in the same article. Categories the employee checks at a single glance may share a cost; the cost condition is then checked for them together, and that has to be written down before the start.
What has to hold for a category to move from stage to stage:
| Number | Stage 1 → stage 2 | Stage 2 → day-to-day work with approval | Later, after go-live |
|---|---|---|---|
| Agent errors (from comparison with the person) | Below the threshold across all proposals since the last change to the agent | — | — |
| Read errors (wrong document) | Below the threshold across all cases with a read since the last change to the agent | Upper bound below the threshold — the approver checks which document the agent used | Still checked in the sample audit |
| Corrections and rejections because of an error (each proposal counted once) | — | Upper bound below the threshold, counted from the last change to the agent | Loosening thresholds from the article on human approval |
| Exceptions | Below the weekly threshold | Below the weekly threshold | Still measured weekly |
| Cost of one case | Model fee below the ceiling | Fee below the ceiling and people’s average time lower than before the pilot | Still measured monthly |
The move from stage 1 to stage 2 is checked more simply — with the raw result across all proposals since the last change to the agent — because in shadow mode the agent executes nothing and stage 2 approves everything anyway. The upper bound is calculated at the decision about day-to-day work. That threshold is above all a usefulness threshold: approving every case limits the risk provided the person really checks the proposal, and if they correct one in three, checking turns into rewriting and the agent makes no sense.
Stop rules: write them down before the first day
A stop rule halts the pilot or one category without a meeting and without waiting for the weekly average. It is written down before the start, together with the name of the person entitled to stop it and the date of a stop drill — a stop nobody has rehearsed often fails when it is needed. The set we start from:
- A write that went through when it should not have. In shadow mode, any change in a system or any message that actually went out stops the whole pilot. It is a permissions failure, not a quality one, and no number of cases outweighs it.
- Data in the wrong case. One customer’s data in another customer’s case stops the pilot until the cause is understood.
- The agent followed an instruction hidden in content. When the agent treated text from an attachment or email as a command — we describe this prompt injection in what an AI agent is — the pilot stops until the safeguards are fixed. This is the last line of defence: the foundation is the lists of permitted tools and destinations, checked before every action.
- Error limit exceeded in a category. The number of errors after which a category returns to stage 1 or leaves the pilot, set in advance for each category.
- Exceptions above the threshold for a full week. A pause and a review of the exception list; the cause usually lies in the data or the process, not the model.
- Cost of a case above the ceiling. A single case above the ceiling is halted and handed to a person with the reason; repeated breaches stop the category until it is fixed.
- Approval time below reading time. When the median time to decide falls below the time needed just to read the proposal, stage 2 data stops being reliable — how to calculate that is in the clicking test.
Three ways a pilot fools its owner
Selected cases. “Clean” cases go into the pilot and the hard ones bypass the agent, because someone wanted the first results to look good. The pilot then measures the easy part of the process. The remedy: the agent gets every case in the period, and if some are deliberately kept out of scope, the report says how many and why.
The approver who stopped reading. The correction rate drops to zero not because the agent got better but because nobody looks any more. The rate alone does not show this — it has to be read together with the median time to decide and the sample audit described in the article on human approval.
Changes to the agent during the pilot. A revised instruction, a different model or a new tool makes a new agent. Cases from before the change say nothing about how the version after it works, so results for the affected categories are counted afresh. Changes are needed in a pilot — that is what it is for — but each one goes into the report with its date, and results are counted separately for each version. The final assessment before the decision is done on one frozen version and a fresh, predeclared period. Tuning the agent after every failed assessment and stopping at the first version that passes raises the chance of a false “go live” just as looking every week does.
The pilot report: one page and a decision for each category
A pilot report does not tell the project’s story. It answers what happens next with each category, and it fits on one page:
- Process and period — what the agent did, from when to when, how many cases came in.
- Agent versions — model, instruction, list of tools and every change with its date.
- Number of cases in each category and at each stage, with the number of cases excluded and why.
- The four numbers for each category next to the thresholds written down before the start, with the upper bound wherever an error rate is counted.
- List of mistakes with their classification: agent, person, both results acceptable.
- Stop rules that fired — what happened, what was fixed, when the pilot resumed.
- Cost of one case next to the cost before the pilot.
- Decision for each category: go live with every case approved; go live and mark as a candidate for approving exceptions after go-live; stays with a person; stop.
- Overall decision — go live or stop — taken jointly by the company and the supplier, with the name of the person on the company’s side who approves it.
Example: an agent that assembles the documents for a complaint
We described an agent that assembles the documents for a complaint in what an AI agent is: it reads the complaint, the order history, the warranty terms and the attachments, and then points out what is missing. The figures below are illustrative — they show how to fill in the protocol and the report and do not come from any implementation.
Assumptions: about 40 complaints a week, a four-week pilot, two weeks in shadow mode and two with approval. Categories: read (finding the order and invoice), a note in the complaint record listing the missing documents (an internal, reversible write) and a request to the customer to supply them (an external action). Reading is assessed separately: the agent records which order and invoice it used, and the approver checks that together with the note — a wrong invoice can, after all, produce the same list of missing documents. The agent prepares a customer request whenever the note lists missing documents, so an omitted request means a note with no gaps listed when there were gaps; the employee corrects that note and it counts as a substantive correction of the note. The decision to accept the complaint does not belong to the agent at all. Thresholds written down before the start by the head of the complaints team:
- agent errors in the note in shadow mode: at most 10 in 100 notes since the last change to the agent (raw result);
- wrong order or invoice read in shadow mode: at most 10 in 100 cases with a read since the last change to the agent (raw result), and in stage 2: upper bound below 10 in 100;
- substantive corrections of the note in stage 2: upper bound below 10 in 100;
- agent errors in the request to the customer in shadow mode: at most 20 in 100 prepared requests since the last change to the agent (raw result);
- substantive corrections of the request to the customer in stage 2: upper bound below 25 in 100 — a usefulness threshold, not an autonomy one;
- an audit of a sample of approved proposals separately for each category — 20 randomly chosen notes (including a check that the agent used the right order and invoice) and 10 randomly chosen requests — done by someone other than the approver: not a single wrongly approved proposal; even one halts the decision on that category until the cause is understood;
- exceptions: at most 15 in 100 cases in any week;
- cost, separately for reading with the note (the employee checks them together, so they share a cost) and for the request: a model fee below the agreed ceiling and employee time lower than before the pilot. Before the pilot, assembling one complaint took 15 minutes on average: about 13 minutes for the assembly itself and about 5 minutes to write the request in the cases that needed one.
The assessment period for the day-to-day-work decision was also fixed in advance: the whole of stage 2, two weeks, with one check of the result after it ended, on the agent version frozen after week 1 — it could not be changed during stage 2.
| Week | Stage | Cases | Exceptions | Notes | Note results | What happened |
|---|---|---|---|---|---|---|
| 1 | Shadow mode | 40 | 7 (18 in 100) | 33 | 26 agreed; 5 agent errors, 2 employee errors | Stop rule: exceptions above the threshold and two breaches of the cost ceiling — the agent looped searching for orders in complaints with no invoice number. At the end of the week, the instruction changed and the limit on actions per case was lowered |
| 2 | Shadow mode | 40 | 5 (13 in 100) | 35 | 32 agreed; 2 agent errors, 1 case with two acceptable results; read wrong in 1 of 35; requests: 1 agent error in 14 | New agent version; counting starts afresh. Agent errors in the note: 2 in 35, about 6 in 100; read errors: 1 in 35, about 3 in 100; request errors: 1 in 14, about 7 in 100 — move to stage 2 |
| 3 | Approval | 40 | 4 (10 in 100) | 36 | 2 substantive corrections | Median time to decide above reading time |
| 4 | Approval | 40 | 5 (13 in 100) | 35 | 0 substantive corrections | No events |
In stage 2 there were 71 notes (80 cases minus 9 exceptions), and 2 of them needed a substantive correction; none was rejected. The observed rate is about 2.8 in 100, but the Wilson upper bound is about 9.7 in 100 — below the threshold of 10, but with no margin. A third such correction in stage 2 would have failed the condition. The read was wrong in 1 of 71 cases (upper bound about 7.6 in 100). The audit found no wrongly approved proposal among either the 20 notes (including any wrong read) or the 10 requests, so the audit condition is met in both categories. We read that result together with the time to decide; it proves neither the quality of the whole category nor that the approver really checked every case. In the second shadow-mode week a third person also reviewed 10 random cases in which the agent and the employee agreed, and all were correct — had they found an error, all 32 would have had to be checked before the decision. The agent prepared the request to the customer in 30 stage-2 cases (outside the exceptions); in 2 of them the employee made a substantive correction (a missing document), in 3 only a stylistic one.
Cost, separately for reading with the note and for the request. The model fee in stage 2 averaged about 60% of the ceiling for reading with the note and about 30% for the request; only two cases in week 1 breached the ceiling, when the loop searching for orders inflated the cost of reading. Employee time in stage 2 from the log, including exceptions:
- reading with the note: 71 checks at an average of 3 minutes and 9 exceptions assembled from scratch at 13 minutes, so 330 minutes for 80 cases — about 4.1 minutes per case against 13 before the pilot;
- request: 30 checks at an average of 2.5 minutes and 4 requests written from scratch in exceptions at 5 minutes, so 95 minutes for 34 requests — about 2.8 minutes per request against 5 before the pilot.
In total 425 minutes for 80 cases, or about 5.3 minutes per case against 15 minutes before the pilot.
The report from this pilot:
| Category | Threshold | Result | Decision |
|---|---|---|---|
| Read: order and invoice | Upper bound of errors below 10 in 100 | 1 in 71, upper bound about 7.6 in 100; cost shared with the note | Go live together with the note |
| Note listing the missing documents | Upper bound of corrections below 10 in 100 | 2 in 71, upper bound about 9.7 in 100; reading with the note: 4.1 minutes per case against 13, fee about 60% of the ceiling | Go live with every case approved; candidate for approving exceptions only once all the conditions in the article on human approval are met in day-to-day work |
| Request to the customer for documents | Upper bound of corrections below 25 in 100 | 2 in 30, upper bound about 21.3 in 100; 2.8 minutes per request against 5, fee about 30% of the ceiling | Go live with every message approved; not a candidate for acting alone, because a message sent to a customer cannot be undone |
| Accepting the complaint | — | — | Outside the agent |
| Overall decision | Exceptions, audit and cost within thresholds from week 2 | An average of about 5.3 employee minutes per case against 15; audit 0 in 20 notes and 0 in 10 requests | Go live — joint decision; approved on the company’s side by the head of the complaints team |
The note’s candidacy does not mean oversight is loosened — it later requires all the conditions in the article on human approval to be met, on cases from day-to-day work. The most valuable line in this report, though, is the week-one row. Had the pilot started straight away with approval, the loop on complaints with no invoice number would have cost employees’ time, not just model fees in shadow mode.
What to do next
Before the pilot starts, write down four things on one page: the categories of the agent’s actions with a threshold for each, the number of cases that threshold demands at your volume, the stop rules with the name of the person who can halt it, and the data arrangements. The agent receives only the data it strictly needs — unnecessary personal and confidential data is removed or excluded before access is granted. Also write down where the data is processed, how long it is kept, who has access to it, when it is deleted, who approves the agent’s proposals and who independently checks a sample. If a category will not gather enough cases in the planned time, you know before the start, not after it. How we run such a project from permissions through the pilot to day-to-day work is described on the AI agents service page.
Describe the process you want to pilot. We will reply with the categories of actions we see in it, how many cases it takes to assess them, and whether a pilot at your volume can settle anything at all.
Sources, as at 12 October 2026
- J. A. Hanley, A. Lippman-Hand, “If Nothing Goes Wrong, Is Everything All Right?”, JAMA 1983 — the rule of three: when no event occurred in a sample of n cases, the upper bound of the one-sided 95% confidence interval for its frequency is about 3/n.
- NIST/SEMATECH e-Handbook of Statistical Methods, 7.2.4.1 “Confidence intervals” — the Wilson confidence interval for a sample proportion; its formula for a two-sided 95% interval (z = 1.96) gives the upper bounds for 2 errors in 100 cases (about 7%), 2 errors in 71 cases (about 9.7%), 1 error in 71 cases (about 7.6%) and 2 errors in 30 cases (about 21.3%). With zero errors the upper bound simplifies to z² ÷ (n + z²), which is where the case counts in the table come from; read 12 October 2026.