Every multilingual voice AI vendor publishes a language list. Forty languages, sixty, sometimes more.
The list is not evidence of anything except intent. What matters is whether the system performs equally well across the languages your callers use — and the published research says it does not.
What the evidence says about language quality
Peer-reviewed audits of automatic speech recognition — the layer that turns speech into text, before any conversation logic runs — show systematic differences:
| Speaker group | Word error rate |
|---|---|
| Well-supported English varieties | ~0.071 |
| Unsupported varieties | ~0.135 (roughly 2×) |
| Speakers whose first language is tonal | ~0.172 (roughly 2.4×) |
Other audits have found error rates as high as 66% for non-native Arabic on a widely used model, 44.2% for Nigerian-accented English in one commercial service, and a 70.6% error rate for female speakers versus 53.5% for male in a model trained on imbalanced data. Age of onset of the language is a significant predictor — people who learned it later are recognised less reliably.
Three practical implications:
Tonal languages are the documented weak spot. That includes Mandarin, Cantonese and Vietnamese — all significant in Canadian and US caller populations.
Accents matter more than language. A system may handle "Spanish" well and handle your callers' Spanish poorly.
Averages hide the problem. A blended accuracy figure across all callers can look respectable while concealing one group being served much worse than another.
The pricing trap
There is a commercial wrinkle worth naming. Some vendors charge per additional language, some include "unlimited languages", and neither pricing model tells you anything about quality.
Worse, a per-language charge creates an incentive to list languages you support thinly: adding a language to the list costs nothing and may justify a higher tier, while supporting it properly costs real work. Treat a long language list with the same suspicion as a long integration logo wall.
A test protocol you can actually run
You do not need a linguistics team. You need real speakers, the same script, and a willingness to score honestly.
1. Pick the languages that matter, not the ones that are available
From your actual call records, list the languages in which you receive meaningful call volume. Test those. A system that handles fourteen languages you never hear is worth less than one that handles the three you hear daily.
2. Recruit real speakers
Five or six people per language, ideally representing the range of accents you actually encounter. A fluent second-language speaker with a heavy accent is a more useful test than a native speaker with broadcast diction. If your callers include people who learned the language later in life, test with them.
3. Write three scenarios, not one
Test at least:
- A routine request — the common call, e.g. booking a standard appointment.
- A specific or unusual request — a detail the system might not expect, e.g. a service it may not offer or a complex rescheduling.
- A frustrated caller — someone speaking quickly, interrupting, and possibly mixing languages.
The third scenario is the one that matters most and the one demos never show.
4. Test code-switching deliberately
Give testers permission to speak naturally, which for many bilingual callers means mixing languages mid-sentence — "I need an appointment for my hijo." Most vendors do not claim to handle it. You should still know what happens, because your callers will do it whether the system is ready or not.
5. Add real conditions
Background noise, a bad connection, a caller interrupting mid-answer, a caller who corrects themselves. These are normal in real calls and absent from scripted tests.
6. Score three things per call
- Did it understand the request? Not "did it respond" — did it understand.
- Was the answer correct? Comprehension and correctness are different failures.
- Did the caller have to repeat themselves? Repetition is the clearest single symptom of an accuracy gap.
7. Measure handoff rate per language after launch
This is the metric to insist on. If one language reaches a human substantially more often than the others, that language is underperforming — and you now have the number to prove it. Ask for it broken out by language, never aggregated, because an aggregate will hide exactly this.
What to ask the vendor
- What is the measured accuracy for each language I need, and how was it measured? A method or a refusal. There is no third answer.
- Is that figure per language, or blended? Blended figures are the standard evasion.
- How do you handle code-switching? "We don't, well" is a legitimate answer. Silence is not.
- What happens when the system mishears? Look for recovery behaviour — asking for clarification rather than guessing.
- Can I test before I commit? Any vendor confident in per-language quality will let you run your own scenarios.
Where machine interpretation must not be used
There is a boundary that testing cannot move.
Healthcare, legal advice, immigration advice and emergencies need a qualified human interpreter. Those are settings where an error is not a frustrated caller but a wrong decision with real consequences — a misunderstood dosage, a missed deadline, a misread condition. Automated interpretation should handle bookings, factual questions and routing, and hand off anything consequential.
If a vendor proposes machine interpretation for those contexts, that is your signal about how they think about risk.
The honest summary
Multilingual voice AI is genuinely useful, and genuinely uneven. It will serve your speakers of well-supported, non-tonal languages well, and it will serve speakers of tonal languages measurably worse — which in Canada and the US includes some of the largest immigrant language groups.
The response is not to avoid it. It is to test per language, measure handoff per language, disclose that callers are speaking to a machine, and keep a human path for anything consequential. A system that does that is a real improvement in access. A system that does not is an advertisement with a phone number attached.
For the wider context on what this means for newcomer service and the language gap in Canada, see multilingual AI voice assistants and newcomer service. For the terminology used above, see the AI voice assistant glossary.