DOI : 10.17577/A product team at a mid-sized insurer chose a speech-to-speech model for its claims hotline largely on the strength of three research papers. More precisely, it chose on the strength of the summaries of those papers. All three reported fast response times and natural turn-taking. The pilot told a different story. On real call center audio with forty simultaneous callers, the model took twice as long to respond and kept talking over customers who interrupted it. The papers had disclosed the testing conditions, but in an appendix that none of the summaries mentioned.
What gets lost when a paper gets compressed
An AI summarizer is good at restating what a paper’s abstract and introduction claim. It is much less reliable with what sits in the experimental setup, the limitations section, and the appendix tables. Unfortunately, that’s where the conditions that decide whether a result applies to you usually live.
Common misses include:
● what hardware the latency was measured on
● whether tests ran one session at a time or under load
● which dataset was used, and whether it was read speech or real conversation
● which model size produced the headline number when a paper reports several
That last one causes trouble more often than you’d expect. A summary can attach the large model’s accuracy to the small model’s speed, producing a system that doesn’t exist.
Summaries are still worth using, mainly for triage. Use them to narrow sixty papers down to the eight worth reading. Then ask specific questions: What hardware was used? How many concurrent sessions? Which datasets? What limitations do the authors list? Ask for page or table numbers with each answer, and check the figures yourself before they go anywhere near a decision document.
Reading S2S benchmarks with suspicion
Speech-to-speech systems are harder to score than text models, because quality has several dimensions at once. S2S benchmarks typically measure some combination of these:
● Response latency
● Naturalness of the generated voice, usually through listener ratings
● Content accuracy, often checked by transcribing the output and comparing it to a reference
● Conversational behavior, such as handling interruptions or following spoken instructions
Each has a weakness. Latency figures often come from a single session on dedicated hardware. Naturalness ratings depend on who did the rating and what they compared against. Accuracy checks that rely on transcription can miss errors that sound fine but carry the wrong meaning, such as a policy number with two digits swapped. Some conversational tests are scored by another AI model, which tends to favor fluent answers over correct ones.
Most published evaluations also use clean audio: studio-quality recordings, one speaker, standard accents. A claims hotline sounds nothing like that. Leaderboard positions can be useful for building a shortlist. Treat them as a starting point, not as evidence the model will work for you.
Build a test set that sounds like your callers
The insurer eventually did what it should have done first. With caller consent, it recorded two hundred short clips covering a realistic mix of conditions:
● Callers on speakerphone in cars
● Background television and crying children
● Regional accents and non-native speakers
● People reading out policy numbers and street addresses
● Callers who interrupted mid-sentence
The team then tested each shortlisted model at the concurrency it expected during a storm surge, not on a quiet Tuesday. It measured time to first audio under that load, how cleanly the model stopped when interrupted, and whether it read numbers back correctly. Human reviewers rated the responses blind, without knowing which model produced which.
The model that ranked third on public leaderboards came out first. It was a little slower in ideal conditions and noticeably steadier when the audio got messy.
Building that set took about two weeks. It can be reused for every model evaluation that follows, which makes it the most valuable research asset the team now owns.
Where the reading time should go
AI can take most of the skimming off a researcher’s plate. That freed time is best spent on the details that change a decision, like the appendix table and the hardware footnote. The insurer’s team didn’t lose weeks because they read too slowly. They lost weeks because they never read the part that applied to them.
