Automatic ticket classification: how to design the categories and test accuracy before you switch it on

Automating ticket handling usually starts with classification: the system reads a message, works out what it is about and routes it to the right person or queue. On our page about customer service automation we say this is the most common first step, because a miss means the ticket lands with someone else in customer service who already has access to it — provided everyone in the customer-service team sees the same tickets. For that step to be genuinely safe, three things have to be designed before go-live: the category list, a queue for tickets the system is unsure about, and an accuracy test. This article covers them in turn. Where this step sits among other automations is shown in our AI automation examples (example 4).

If a vendor quotes an accuracy percentage, ask which tickets it was measured on and how it was calculated. You will not find such a percentage here. The number that matters is produced on your own tickets, in the spreadsheet described below — and it is different for every category.

How it works, in one paragraph

A ticket arrives by email, from a form or from a chat. The system — in practice a language model, meaning a program that understands text and can assign it a category from a list, or a simpler model trained on your own examples — picks one category and gives a number describing how confident it is. If the confidence is at least the threshold, the ticket goes to that category’s queue. If it is below, the ticket goes to the “manual review” queue, where a person assigns the category. If a system you are considering does not give such a number, ask the vendor how it flags tickets it is unsure about — without that, at most whole categories can go to the manual queue, not individual doubtful tickets. Priority and handling deadlines are separate decisions, which we come back to at the end. How such flows are built in an automation tool is covered in our n8n guide; here we deal with what has to be settled before anything is built.

When automatic classification is not worth it

Let us start with the cases where the answer is “not now”:

• There are so few tickets that one person reads them all anyway. Automation saves them a few clicks a day, and accuracy will be hard to assess reliably until enough tickets have been collected.
• A simple rule is enough. If the category is determined by the address the message was sent to, a drop-down in the form or the sender, a rule in the mailbox or ticketing system may be enough — without a model. Just check the exceptions, such as customers who write to the wrong address. Start with rules like that and leave the model for the rest.
• The categories change every month. Every change to the list invalidates the test, and it has to be repeated.
• Nobody owns the manual queue. Without that person, uncertain tickets wait longer than they did before automation.
• Tickets arrive mainly by phone, without text. Speech-to-text has to be solved first, and that is a separate problem with its own errors.
• A miss costs more than a handover and a delay. Formal complaints with deadlines set by law or contract, reports of faults that threaten safety, legal matters — those categories stay with a person or reach a person through a fixed rule, whatever the model says.

The category list: from real tickets, not from the org chart

The category list is built from a sample of real tickets. We suggest drawing about two hundred cases at random from the last three months across all written channels. The unit is a case, not a message: if the export has a separate row for every message, first keep one row per case, otherwise long threads would land in the sample more often. Then, in Excel, add a column with the RAND function, copy it and paste it back as values (because RAND draws again every time the sheet recalculates), sort, and take the first two hundred rows. If there were fewer cases, take them all. Then show the system only what was known at the moment a ticket was first routed: the first message and the form fields, without later replies, because those often give the right category away.

Before assigning categories to the tickets in the sample, look through a few dozen tickets from outside it and write down categories according to who handles a case and how, not according to the product catalogue or the company’s departments. The test question is: do two tickets from the same category go to the same person and get handled in a similar way? Add an “other” category to the list. Decide straight away, too, what happens to a ticket the system marks as “other”: we suggest sending it to the manual queue, because “other” has no person of its own.

Why eight to fifteen, not forty

As a starting point we suggest eight to fifteen categories. That is a starting point, not a rule or the output of a calculation, but there is simple arithmetic behind it:

• in a category that five tickets belong to, one mistake moves its result by 20 percentage points;
• in a category with twenty tickets, one mistake is 5 percentage points.

With two hundred tickets and forty categories there are five tickets per category on average; with ten categories, twenty on average. Even twenty is not much. If the system correctly recognised 18 out of 20 tickets, the result is 90%, but with so few tickets the true accuracy could lie roughly between 70% and 97% (that is the 95% confidence interval calculated with the Wilson method described in the NIST statistics handbook — the source is listed at the end). So a test like this shows clear problems, not differences of a few points. The more tickets a month, the longer the list that can be tested; with low volumes, results for rare categories have to be collected over a longer period.

The average is not the whole story: rare categories will get only a few tickets in the sample. If you want the system to route them automatically, add extra examples for them from older months — provided they are handled the same way as today. Use those examples only to check how many real tickets in that category the system recognises and what it confuses them with. Evaluate them in a separate copy of the workbook, and calculate precision and the shares of categories on the random sample only, because the added examples change the proportions between categories.

Where people assign different categories

Ask a second person from customer service to assign categories independently to a few dozen tickets from the sample. Where two people choose different categories, look at the reason. Sometimes a ticket is about two things at once — then assign the category of the person who should deal with it first. The cause can also be an unclear definition: add one sentence and an example to it, or merge the two categories into one. Microsoft gives similar advice in the documentation of its text-classification service: if the model keeps confusing two categories, it is worth considering merging them.

Fixed rules for categories where a mistake is not acceptable

The categories mentioned among the “not now” cases — formal complaints with a deadline, faults, legal matters — should, to begin with, be recognised by fixed rules: by a form field, an address or specific words. A ticket that matches the rule goes to a person regardless of what the model says. For cases where a mistake is not acceptable, the rule has to rest on a signal that cannot be bypassed: a mandatory form field or a separate address that customers are actually required to use. A rule based on words can help, but it is not enough — missing nothing in the test does not guarantee it will catch the next such case. If there is no such signal, the tickets that could include such cases stay with a person. Check every rule on the same sample too: how many such cases it caught, and how many it missed because the customer described them in other words. That is a separate test: the person assigning the reference marks whether the ticket should have fallen under the rule, and you then compare that with whether the rule fired — most simply in a separate copy of the workbook, in the “Notes” column.

The person’s label as the reference

Now the person who handles tickets day to day assigns the correct category to every ticket in the sample. This is the reference the system will be compared with, so three principles apply: categories are assigned before seeing the system’s answer, the category list used is exactly the one the system will get, and only what the system will also get is considered — the first message and the form fields, without the customer’s later replies. At a minute per ticket, two hundred tickets take 3 hours and 20 minutes; work it out for yourself, because long first messages take longer.

Download the spreadsheet template (Excel). Use one copy of the file for the adjustment sample and a separate one for the test on new tickets described below. The “Sample” sheet holds up to a thousand tickets and has these columns:

On the “Categories” sheet you enter your category list and, once set, a confidence threshold for each. The workbook calculates, for every category, the precision and recall described below, as well as the number of tickets routed automatically and handed to manual review. If the system gives no confidence, enter 1 for every ticket (a convention, not a measured confidence): categories routed automatically get a threshold of 1, and the rest stay without a threshold and go to manual review as a whole. Keep the list matching row numbers to ticket numbers in your system separately, outside the sheet. Do not copy names, addresses, order numbers or message text into the sheet, because the columns do not need them. The content of the tickets the system will classify usually includes customers’ personal data. The rules for passing it to a language model, including what follows from the GDPR (the General Data Protection Regulation), are described on our security and GDPR page.

Two stages of the test: the sample, then new tickets

The general principle of running a new system alongside current work before it replaces it is covered in the sixth step of our article on how to implement AI in a company. For ticket classification one distinction matters: the sample on which you improve the category list and set the thresholds cannot be the one on which you then check the result. Anyone who repeatedly adjusts settings to one sample gets a result fitted to that sample — which is why Google’s machine-learning course recommends a final check on data that has not been used before. In practice:

1. The sample is for adjustments. The system classifies the same two hundred tickets the person labelled. Using the results, you improve the category definitions, merge confused categories and set the thresholds. After every rename, merge or definition change that moves tickets between categories, reassign the affected sample tickets — the workbook shows how many labels are no longer on the list. After every change to category definitions or system settings, run the system over the whole sample again, because its earlier answers used the old version. Set thresholds only on the new results. No ticket is routed anywhere automatically yet.
2. The settings are frozen. The category list, rules, system settings and thresholds are written down and do not change until the end of the test.
3. New tickets are for the decision. The system classifies live tickets, but a person still routes them, as today, without seeing the system’s suggestion. What is recorded is the category the system chose, its confidence and the routing that the frozen thresholds would produce, alongside the first category the routing person assigned, before any correction. People make mistakes too, so the results are calculated on a random sample of the new tickets, to which the manual-queue owner assigns a category independently — from the first message, the form fields and the category definitions, without looking at the system’s or the routing person’s answer. Tickets where the system and the routing person chose different categories are worth reviewing separately — only after the sample has been labelled — to understand the mistakes, but only those that fell into the random sample count towards the results. The reference is the queue owner’s category (or, if the owner routed that ticket, the category of someone who did not route it) for every ticket in the sample — a disagreement with the routing person changes nothing in it, because it is the routing person’s accuracy being measured. If you want a more reliable reference, have one more person assign a second category independently, also without seeing the system or the routing person, and let a third person settle disagreements between those two. The test lasts until every category you want to switch on has collected, within that random sample, the ticket counts written down before the test started, or until the maximum test duration written down beforehand — whichever comes first (more on that below). The decision to switch a category on is made on the results of this stage. In the workbook the final agreed reference goes in the “Category — reference” column and the routing person’s original category in a separate column, so both the system’s result and people’s accuracy today stay visible.
4. People’s mistakes today. During the same period, count how often a ticket already reaches the wrong person today, and how long it takes to reach the right one. People make mistakes too, and without those numbers there will be nothing to compare the post-launch result with.

Two numbers for every category

A single overall figure — “the system is right 80% of the time” — says little, because it hides the categories where the system is wrong most often. So two numbers are calculated for every category. First in plain words, using complaints as the example:

• Of everything the system routed to complaints, how many really were complaints? This number is called precision. Low precision means the person handling complaints receives other people’s tickets.
• Of all the real complaints, how many did the system route automatically to the right queue? This number is called recall. Low recall means some complaints ended up somewhere else or in the manual queue.

The formulas, in the same form as in Microsoft’s documentation and Google’s machine-learning course:

• precision = tickets correctly routed to a category ÷ all tickets the system routed to that category;
• recall = tickets correctly routed to a category ÷ all tickets that, according to the reference, belong to that category.

Both numbers are read from a table of mistakes (a confusion matrix): the rows are the reference category, the columns are the routing the system chose once thresholds are applied. Correct routings sit on the diagonal and mistakes show up immediately off it — including which category the system most often confuses a given one with. The “manual review” column holds tickets handed to a person: they are not routing mistakes, but they lower recall, because the system did not route them automatically. In Excel you can build it with a PivotTable: “Category — reference” in rows, “Routing” in columns, the count of “Category — reference” in values; filter out the empty “(blank)” rows, because the workbook has room for a thousand tickets. A single number can also be calculated with the COUNTIFS function — for example, the number of rows where the reference category is “complaint” and the routing is “complaint”, divided by the number of rows routed to “complaint”.

Example: two hundred tickets, four categories

The numbers below are made up to show the arithmetic. They are not a real company’s results or a benchmark. The sample has two hundred tickets, the thresholds have already been set, and tickets the system marks as “other” go, as we suggested, to the manual queue.

From this table:

“Other” has no precision of its own here, because it has no queue of its own. What matters is something else: sixteen tickets that really are “other” were routed by the system to specific people. Those are worth looking at first.

In total the system would route 132 of the 200 tickets automatically, 111 of them correctly, or about 84%, and 21 to the wrong person. Another 68 tickets would go to the manual queue, 54 of which really are “other”. That single number — 84% — does not tell you that invoices have a precision of 60%, or that two complaints landed with the person handling order status. Also remember the arithmetic from the start: with categories of twenty or thirty tickets, individual mistakes move the result a lot.

The confidence threshold and the “manual review” queue

The number describing the system’s confidence does not necessarily mean anything — you have to check it on the sample. Take the rows where the system chose a given category, sort them by confidence and see where the mistakes cluster. In the workbook, filter the “Category — system” column, copy the visible rows, paste them as values into a new sheet and sort them there — the sample sheet is protected so its formulas cannot be overwritten. If there are almost no mistakes at high confidence and many at low confidence, a threshold makes sense. You set it where the mistakes above the threshold are acceptable given the cost of a miss in that category. If mistakes are spread evenly across all confidence levels, a threshold will not help, and the category stays with a person.

The threshold is set for each category separately. It works in one direction: the higher the threshold, the more tickets go to the manual queue and the fewer mistakes get through. For a category where a miss means only a handover within the team, the threshold can be lower than for a category where a miss delays something important.

The “manual review” queue is part of the design, not a dumping ground. It needs a named owner, a deputy and a handling deadline — like any queue of cases waiting for a human decision, which we cover in our article on human approval. Track its size as well: if most tickets end up there, automation changes little, and if its owner cannot keep up, uncertain tickets wait longer than they do today.

The decision: which categories to route automatically

Before you see the results from new tickets, write down three things for each category: the minimum precision you will accept, the minimum number of tickets the system must route to that category before you judge its precision, and the minimum number of tickets that belong to it according to the reference before you judge its recall. The test lasts until both ticket-count minima are reached — for precision and for recall. Write down a maximum test duration too — the test ends at whichever comes first. A category that has not collected the required tickets by then (for example because the system hardly ever picks it, or sends it all to manual review) stays with a person. If a category has to pay off, not just do no harm, write down a minimum recall as well. Low recall has two causes. Tickets sent to the manual queue do no harm as long as that queue keeps to the agreed handling deadline — but then automation takes over little work. Tickets routed automatically to another category are misroutes and do harm: they lower that category’s precision and show up in the table of mistakes. Writing these conditions down beforehand protects you from fitting the requirements to the result you happened to get. Keep in mind that precision and recall have different bases: precision is calculated from the tickets routed to a category, recall from the tickets that really belong to it. A category can have many real tickets but few routed to it; if the system routed none, precision simply cannot be calculated — it is not 0%. With few tickets the result carries a lot of uncertainty, as in the 18-out-of-20 example. The minimum precision depends on the cost of a miss, so we do not set it for you. A category is switched on only if the shared conditions are met as well: the people it routes to have access to those tickets, and the manual queue is staffed for the tickets that will flow into it. The decision then looks like this:

In the first version, a person corrects the misses. Whoever receives a ticket that is not theirs sends it back to the manual queue, and the queue owner assigns it to the right person. Make sending it back easy — one button or an agreed label — or misrouted tickets will be passed from hand to hand without a trace. When a category can be left without constant checking, and after which event checking comes back, is covered in our article on human approval — those rules are applied to each category separately.

What a miss costs and how to measure it

The service page says a misrouted ticket lands with someone else in customer service who already has access to it. That is true if everyone in the customer-service team sees the same tickets — check it in the permissions before assuming it. The cost is then extra handovers — to the manual queue and on to the right person — and a delay. That cost is measured in three places:

• in the test on new tickets — how many of the tickets the system would have routed automatically had a different category from the reference. These are possible misses, not actual ones, because the routing person was still routing the tickets;
• in today’s work — how often a person routes a ticket to the wrong person, and how long it takes to reach the right one;
• after the first categories go live — what share of automatically routed tickets came back to the manual queue or were given a different category by a person (the return and reclassification rate, from which the misses are separated — see below), and how long it took from arrival until the ticket reached the person who finally handled it.

For every return or category change, record the reason. Count as a classification miss a wrong routing based on the information available at the moment of routing; mark separately changes caused by new information from the customer or by how the work is organised. Do not count tickets sent straight to the manual queue as misses: there a person assigns the category, by design. Do count tickets routed automatically to the wrong person, even if they later came back to that queue — each one once, however many times it came back. And compare numbers on the same basis. Calculate the miss rate among automatically routed tickets separately, and — before and after go-live — the miss rate among all tickets belonging to the categories switched on, including those handled manually. Alongside them, report the manual-queue share and the time to reach the right person across all tickets — otherwise the result improves on paper only because hard tickets moved to the manual queue.

There is one exception to keep in mind when designing the queues. A handover between people in customer service stays within the circle of people who have access to the tickets. But if a category routes tickets to another department — finance, HR, lawyers — a miss can show a customer’s data to someone who should not see it. Leave that routing to a person at first, or send it to the manual queue. Be especially careful with tickets in which customers themselves describe, for example, their health; there is more on this on our security and GDPR page.

What to record for every ticket

Without a record you will not be able to calculate any of the numbers above after go-live. A few fields per ticket are enough:

• channel and time of arrival;
• the category chosen by the system and its confidence;
• the version of the category list, rules, thresholds and system settings the ticket was classified with;
• the queue the ticket went to, and when it was routed;
• whether and when an automatically routed ticket came back to the manual queue;
• the category finally assigned by a person, who changed it, when and why (a miss, new information from the customer, or how the work is organised);
• when the ticket reached the person who finally handled it.

The ticket’s content is already in the ticketing system — there is no need to duplicate it in the classification record.

Model-assigned priority: not in the first version

On the service page we list assigning priority together with recognising and routing tickets, because it is one step of the process. In the first version, though, priority is set by fixed rules, not by the model. Model-assigned priority, meaning recognising urgency, looks like a small add-on, but it is a second classification, with its own list, its own test and two different kinds of mistake: an urgent ticket mistaken for a routine one waits too long, while a routine one mistaken for an urgent one takes up time needed for the genuinely urgent cases. So categories go live first. Cases that cannot wait are, to begin with, recognised by fixed rules — by sender, a form field or specific words — and sent straight to a person. Check those rules on the sample too, because they can miss an urgent case described in other words. Without a signal that cannot be bypassed, such as a mandatory field, urgent cases stay with a person. The handling deadlines that apply today stay as they are, including for the manual queue — in the first version the model neither sets nor changes them. Model-based priority comes only once the categories work and there is a separate sample for it.

What next

Draw a sample, write down the categories and ask someone from customer service to assign them to two hundred tickets. There is no system yet, and already at this stage you will see which categories people confuse and how many tickets end up in “other”. Only then is it worth talking to a vendor and asking them to classify the same sample. Before any ticket reaches their model — even one run “in your environment”, because the model may still run on the vendor’s side — settle which fields leave the company, who receives them, in which region they are processed, on what legal basis, and whether they may be used to train the model. Until that is settled, do not hand over real tickets, even with names and addresses removed: the message text can still point to a specific person or contain confidential company information. At this stage, sample tickets you write yourselves, modelled on real ones, are enough. The questions to ask a vendor are on our security and GDPR page. How we at ploch.ai approach automating one process, from measurement through pilot to upkeep, is described on our page about AI automation for business.

Sources, as at 12 October 2026

• Microsoft Learn — “Custom text classification model concepts” — formulas for precision and recall calculated for each category separately, the confusion matrix, and the advice to consider merging categories the model keeps confusing; an example of technical support tickets where a mistake routes a ticket to the wrong team; accessed 12 October 2026.
• Google for Developers — “Classification: Accuracy, recall, precision, and related metrics” — definitions of precision and recall, and why overall accuracy alone misleads when one class is rare; accessed 12 October 2026.
• Google for Developers — “Thresholds and the confusion matrix” — the threshold that turns a model’s score into a decision, and the confusion matrix; accessed 12 October 2026.
• Google for Developers — “Datasets: Dividing the original dataset” — a final check of the model on a separate dataset, and how a dataset used repeatedly for adjustments “wears out”; accessed 12 October 2026.
• NIST/SEMATECH e-Handbook of Statistical Methods, 7.2.4.1 “Confidence intervals” — the confidence interval for a proportion in a sample, including the Wilson method recommended for small samples. For 18 hits out of 20 tickets the 95% interval is roughly 70% to 97%; accessed 12 October 2026.
• Microsoft — RAND function, COUNTIFS function and creating a PivotTable in Excel; accessed 12 October 2026.