Many evaluations of enterprise voice AI still start with a spreadsheet. Word error rate sits in one column, and a mean opinion score from a listening panel sits in the next, often measured on fixed test sets under controlled conditions. None of those numbers are captured over a phone line, and the polished demo that usually seals the deal happens in a quiet room.
Once it’s deployed, the same agent has to actually perform, whether it’s booking an appointment or resolving a billing dispute over a real phone line. The audio an agent receives might be compressed or choppy from packet loss, often from a speakerphone in a car or a busy kitchen. The caller may have an accent that the test set didn’t cover, and they may start talking before the agent finishes its sentence.
I see this play out constantly in my work building voice infrastructure for enterprises. A system that ranks well on a public leaderboard can still struggle to resolve calls on its own in its first weeks of production, and the metrics used to select it rarely explain why. Every call it fails to resolve lands back in the most expensive channel: 2022 Gartner research puts assisted service at $14 per contact, compared with $0.09 for digital self-service.
That makes choosing a benchmark a business decision, not just an engineering one. If the goal is to predict whether an agent will pay for itself in production, the evaluation needs to move from “How accurate is this model?” to “Can this system reliably complete a conversation?”
Word error rate is only as good as the audio it’s measured on
Word error rate (WER) counts how many words a speech recognition model gets wrong against a reference transcript, and it’s a useful starting point for judging one. The headline figures vendors quote usually come from read-speech test sets: high-quality recordings of people reading prepared text. Contact center audio is lower-fidelity phone audio, full of crosstalk and false starts.
Telephone and conversational benchmarks do exist, including Switchboard, Fisher and CallHome. But Switchboard and Fisher were recorded by strangers paired to talk about assigned topics rather than customers calling with a problem, most of these corpora are more than twenty years old, and none of them carry your codec, your accent mix or your domain vocabulary. A good score on them is evidence, not proof.
WER also weights every word the same. A dropped article costs as much as a misheard digit in an account number, and only one of those breaks the transaction. For agents that collect names, dates, amounts and account numbers, accuracy on those specific details decides whether the booking or payment goes through, which is why teams that care about outcomes track entity or slot error rates alongside WER.
A peer-reviewed study published at EMNLP 2020 tested this directly. Researchers from Avaya and three universities ran three commercial speech recognition systems over 50 real call center conversations spanning booking, finance, insurance and telecom. Error rates ranged from roughly 14% to 23% by domain, against the 2-3% often cited on standard datasets, and were highest in booking and telecom calls, which the authors linked to more mentions of dates, times, money and product names.
The same study shows that even a phone benchmark can flatter a system. The three systems scored around 10% to 12% on Switchboard, and their overall error rate on the real call center audio was roughly 60% higher.
Recognition models can also produce plausible text the speaker never said, especially when the audio is clipped or cut off. A 2024 study of one widely used open-source model found that roughly 1% of transcriptions contained entire phrases or sentences that didn’t exist in the audio. In a voice agent, those made-up words pass straight to the language model, which has no signal that they’re wrong and will act on them.
Reported scores are also sensitive to preprocessing. Evaluation pipelines normalize text differently before scoring, and audio preparation such as loudness handling varies between setups, so two vendors can publish numbers on the same dataset that aren’t comparable.
Listening tests don’t capture interaction quality
Mean opinion score (MOS) has a similar blind spot. The method traces back to the International Telecommunication Union’s P.800 recommendation, approved in 1996 to rate the transmission quality of phone connections.
MOS measures whatever listeners are asked to rate, and P.800 itself describes conversational tests alongside listening-only ones. The limitation is in what the industry publishes. The MOS figures in model cards and vendor comparisons are usually short-clip naturalness scores: listeners hear a few seconds of synthetic speech and rate how natural it sounds.
That captures how a voice sounds in isolation, not anything interactive, such as how quickly the agent responds after the caller finishes or whether the agent stops speaking when interrupted.
A voice with a high score can still sit on top of a slow system, or one that mistakes its own voice echoing down the line for the caller cutting in. On a live call, that shows up as callers repeating themselves, talking over the agent, or hanging up. None of it appears in an offline listening test.
Component benchmarks don’t add up to system performance
Most production voice agents run as a cascade of speech recognition, a language model, and speech synthesis. That design works well when the pieces are built to work together, but many enterprise stacks are stitched together from separately built vendors, each benchmarked on its own. At every handoff, context the next stage needs, such as the caller’s tone, a hesitation, or the timing of a pause, can be lost or delayed. Systems where components share context across handoffs are less exposed to this, which is one reason the system matters more than any single component’s score.
Latency compounds the same way. Vendors quote response times for their own piece, usually measured on a single request under ideal conditions. Three components that each respond in 200 milliseconds look fast on paper, but stacked together they already add up to more than half a second. Callers hear the sum of every stage, plus the time the system waits to decide the caller has finished, any lookups in back-end systems, and the trip across the phone network.
Concurrency matters as well. A speech model that responds quickly on a single test call can slow down considerably when it’s handling hundreds at once.
This is also a hidden cost in build-versus-buy decisions. A team that assembles its own stack from separate components inherits the integration work, including the latency budget and the debugging when tone or timing gets lost between vendors. That can be the right call for teams with deep speech expertise, but the total cost should include that engineering, not just per-minute pricing.
Five measurements that predict how an agent performs on real calls
Better evaluation measures the complete system on audio that resembles real traffic, whatever architecture sits underneath. Each of the five measurements below ties back to a number operators already track, such as containment, handle time, or customer satisfaction.
Task completion rate
Start with the outcome the agent was deployed to deliver. Did the appointment land in the scheduling system with the right date, and did the refund reach the right account? Score it against the system of record rather than the transcript, by checking what the agent actually entered into the booking or billing system. This is the number closest to containment, and it’s the one the business case rests on.
Recovery after interruption
Have testers interrupt the agent mid-response and reverse a request they made earlier in the call. Then check whether the agent stops talking promptly and acts on the corrected request instead of the original one. Log how long it keeps talking after the caller starts speaking, and whether the abandoned request leaves anything half-finished, like a partly submitted form.
Escalation and handoff quality
Some calls should go to a person, and the agent has to recognize which ones. Measure both ways it can get this wrong: keeping a call it can’t resolve, which frustrates callers, and transferring one it could have finished, which erodes the business case for the deployment. Then check the handoff itself, including whether the human agent receives the transcript, the details already collected, and the reason for the transfer, so the caller doesn’t have to start over.
Turn-taking
Most turn-taking failures come down to how the agent decides a caller has finished speaking. Measure how often the agent talks over the caller and how often it leaves a gap long enough that the caller asks whether anyone is still there. Settings tuned on clean audio behave differently on a noisy line, where background sound can hide the end of a turn or register as speech that never happened.
End-to-end latency
Measure from the caller’s last word to the agent’s first audible word on a real phone call, so the number includes the delays the phone network adds. Report the typical response time alongside the slowest 5% of turns, because a single long freeze can undo a call that was otherwise going well. Run the test at the call volume expected at peak, since latency often climbs as load grows.
Build the test set from real calls
All five measurements are only as good as the audio they run against. Where privacy and compliance rules allow, sample real recordings and deliberately keep the difficult ones: interruptions, crosstalk, heavy background noise, weak connections, and the full range of accents your callers actually have. Weight the sample toward the call types that carry the most revenue or risk, such as payments and cancellations.
Where real recordings can’t be used, reproduce those conditions on purpose. Degrade clean audio to phone quality, add background noise and dropped audio at realistic levels, and script testers to talk over the agent rather than waiting politely for it to finish.
What to ask vendors before signing
These measurements also make for a sharper procurement process. Instead of asking for a leaderboard rank, ask every vendor the same questions. Will you run a pilot on our own recorded or simulated calls and report task completion, escalation and handoff quality, and the slowest 5% of response times at our expected peak volume? How was each published WER or MOS figure produced, on what dataset, with what audio preparation, and was it measured over a phone line?
Then ask who owns the whole call. If speech recognition, the language model, and speech synthesis come from different providers, who is accountable for end-to-end latency and for calls that fail between components? A vendor that can’t answer with numbers from realistic traffic is leaving that risk with the buyer.
Make system-level testing the default
None of this means abandoning WER, MOS or component benchmarks. Measured on real phone audio, they remain useful for catching regressions and comparing components under controlled conditions. They just can’t stand in for an end-to-end answer to the question that matters in production: does the agent complete the call?
The teams I see getting voice agents into production fastest treat this as a systems problem from day one. They run their evaluations on recorded or simulated phone traffic and track task completion, interruption recovery, escalation and handoff quality, turn-taking errors and tail latency on every release, the way they’d track error rates on any other production service rather than as model scores in a spreadsheet.
The same lens applies to investors and operators sizing up the voice AI market. Benchmark wins are easy to announce, but resolved calls and lower cost per contact are what show up in renewals. Companies that measure what callers actually experience, and can show those numbers to customers, are better placed to turn pilots into long-term contracts.
