Answering machine detection accuracy: our measured data
Answering machine detection accuracy is two numbers, not one: how many live people reach agents, and how many calls sent to agents are really people. Here are both for nomoVM, measured on hand-labelled calls the model never saw, with the errors that remain.
of live people sent to agents
of agent calls were live people
of calls listened to 3.2 s instead of 2.0 s
hand-labelled test calls, unseen in training
How we measured
- The calls. 400 answered outbound calls from a real working day on a VICIdial cluster, with rates weighted back to the day's real mix of humans and machines.
- The labels. A person listened to each call and labelled it: human, machine, screening, silence or tone. Recordings that were the same audio as calls to other numbers were labelled machine. We do not use the dialer's own dispositions or another AMD's verdicts as truth: when we checked, about 25% of calls another AMD called "human" were machines.
- The model. The model had never seen these calls in training.
- The rules. Everything that runs in production: tone detection, the repeat-recording check and the adaptive window.
Results
| Setup | Live people sent to agents |
|---|---|
| nomoVM, 2.0 s window with adaptive 3.2 s for unclear calls | 94.9% |
| nomoVM, fixed 2.0 s window | 89.5% |
| Hosted transcript-based AMD used before, same kind of traffic | about 80% (61% on another day) |
Of the calls nomoVM sent to agents, 88% were live people. On a day of about 135,000 answered calls, that still means about 1,650 machines reaching agents: mostly short personal greetings and recordings designed to sound human.
Speed
- Most verdicts arrive 2.0 seconds after answer, once 2 seconds of audio have been heard.
- Unclear calls, 5 to 9% of the total, get up to 3.2 seconds.
- Calls that end early are decided at once with the audio that arrived.
- On a single CPU core, the smaller model handled 200 calls at the same time with verdicts at 2.00 s (median) and 2.38 s (95th percentile).
What still goes wrong
- About 5 in 100 live people are judged machines. Most are very quiet answers, people talking to someone else in the room, or noisy lines.
- About 12 in 100 agent calls are machines: short personal greetings, and new recordings built to sound like a person, until they have been heard on enough numbers to be fingerprinted.
- Ringback, busy, SIT and fax are almost never wrong: they are fixed tones. In about 3,800 labelled calls, ringback never came before a live answer.
Caveats
These numbers come from one center's traffic. Lists, regions, carriers and greeting styles differ, and your results will too. That is why the free plan includes 30,000 calls a month: run it on one campaign, label a few hundred calls yourself with our testing guide, and compare with what you have now. We'll update this page as we measure more traffic.
Stop sending voicemail to your agents
30,000 calls a month free. No card, no setup fee. Your dialer falls back to Asterisk AMD() on its own if anything goes wrong.