AI voice agents
How to test an AI voice agent before it handles real calls
A pleasant demo call is not a launch test. A business voice agent must understand the caller, follow the agreed boundary, complete or refuse the right action, and leave the business system in the correct state. Test those layers separately and together, with expected results written before each call. That turns subjective feedback such as “it sounded good” into a release decision your team can defend.
By SVS Digital · Published
Start here
Before launch, define the small set of calls the agent may handle and create a pass-or-fail matrix for each one. Check four things: the spoken conversation, the business decision, the actual calendar or CRM result, and the phone path when something fails. Include unclear speech, corrections, names, numbers, noise, interruptions and unavailable integrations. Repeat critical scenarios, have someone outside the build team listen, and do not open real traffic while a critical action can be wrong, duplicated or falsely confirmed.
1. Turn the launch scope into observable pass conditions
Write the first version's allowed jobs in plain language. For example: answer approved questions, capture a service enquiry, record a callback request, or request an appointment that a person confirms later. “Handle reception” is too broad. Each job needs an expected end state, a prohibited action and an owner who decides whether the test passed.
NIST's AI RMF Playbook recommends choosing testing procedures and metrics that show whether a system is fit for its purpose, defining acceptable limits, documenting outcomes and correcting the course when limits are exceeded. For a voice agent, a useful release gate is concrete: every critical scenario must pass, no test may create an unauthorized real record, and known non-critical defects must have an owner and fallback. This is an operational test plan, not a guarantee of legal or sector-specific compliance.
- Allowed outcome: what may the agent say, collect or change?
- Expected system state: what should exist after the call?
- Forbidden outcome: what must never be promised or recorded?
- Fallback: what does the caller hear when the task cannot be completed?
- Release owner: who may accept, reject or narrow the launch?
2. Test four layers instead of grading only the transcript
Separate the test result into conversation, decision, action and telephony. The conversation layer asks whether the caller could understand and interrupt the agent. The decision layer asks whether it stayed within scope and chose the correct next step. The action layer verifies the calendar entry, CRM record, message or callback task. The telephony layer covers connection, audio, transfer and disconnection behaviour.
OpenAI's official voice-agent guidance says to test both the conversation and the completed task: for an appointment, listen to the confirmation and separately check that the correct appointment was saved. It also recommends recording the audio, events, tool results and application state needed to verify an outcome. A fluent confirmation is therefore a failure when the booking is missing, duplicated or stored for the wrong time.
- Conversation: intelligible, concise and able to recover after a correction.
- Decision: correct route, boundary and request for human help.
- Action: exact fields, permissions and final state in the connected system.
- Telephony: usable audio, clean ending and tested failure path.
3. Build a matrix from real calls and awkward variations
Start with five to ten frequent call reasons, not a list of every possible phrase. Give each row a caller goal, starting conditions, facts the agent may use, expected questions, expected action and pass condition. Then add variations: incomplete information, two intents in one call, a changed request, an unavailable slot, a caller who asks for a person, and a request outside the agreed service.
Voice adds errors that a text test will miss. Test family names, company names, street names, dates, times, email addresses and phone numbers with different speakers. Add background noise, a quiet speaker, a long pause, an interruption and a correction such as “not fifteen, fifty.” OpenAI recommends checking accents, background noise, language switches, names and numbers, and judging recognition separately from output intelligibility. Use representative voices without claiming that a small test covers every caller.
- Ordinary call with all required information.
- Missing or ambiguous detail that requires one clear question.
- Caller corrects a name, number, date or requested service.
- Noise, pause or interruption changes turn-taking.
- Unavailable action, out-of-scope request or explicit request for a person.
4. Verify tools with controlled data before using live records
Use test calendars, test contacts and recognisable reference values. Check what happens when a lookup returns no result, a calendar times out, a record already exists or a notification cannot be sent. The agent should not convert a failed action into a spoken success. For every write, compare the caller's request, tool arguments, returned result and final business record.
ElevenLabs documents separate tests for a next reply, a full multi-turn simulation and a tool call with expected parameters. Its simulation tests can mock selected tools so controlled responses are returned instead of calling live systems. Where the chosen platform cannot isolate a side effect, disable that action and test the fallback first. A production write should be tested only in an agreed, controlled window with a cleanup plan.
- No live customer is contacted by a test.
- A failed tool produces an accurate fallback, not a confirmation.
- Repeated wording does not create a duplicate booking or lead.
- Stored names, numbers, consent fields and notes match the confirmed facts.
- The team receives the summary through the channel it will actually monitor.
5. Listen for timing problems and diagnose the right layer
A correct transcript can still be a poor call. Listen for the delay before a useful answer, the agent speaking over the caller, long silence, rushed numbers and awkward recovery after an interruption. Record whether the problem is recognition, business logic, a slow tool, speech generation, the phone network or playback. Change one factor at a time before comparing another run.
Twilio's Conversation Relay call summary illustrates useful diagnostic signals: interruptions by the caller or agent, silence, response-time components and a chronological event timeline. Twilio also notes that its latency view does not include every last-mile carrier or internet delay. Use available provider metrics to locate a problem, but pair them with human listening on real phone paths; a single average cannot decide whether the conversation is usable.
- Time to the first useful spoken answer, not merely an acknowledgement.
- Caller can interrupt and correct without losing the task.
- Names, numbers and confirmations are spoken at a checkable pace.
- Silence and overlap are investigated with audio and event data.
- Tests include more than one phone, network or caller where practical.
6. Repeat critical tests, launch narrowly and keep a regression set
Generated conversations vary, so one pass is evidence that the agent can succeed, not that it will always succeed. ElevenLabs' testing documentation supports repeated runs and groups results by failure reason. Repeat high-consequence scenarios and every known flaky case. Ask an employee who did not configure the agent to try natural wording and score the same written conditions.
When the gate passes, start with limited call reasons, a staffed fallback and a named person reviewing early calls. Turn real failures into new regression tests before changing the prompt or tools. Pause or narrow the pilot when the agent confirms an action that did not happen, performs an unauthorized action, loses essential caller details, repeatedly blocks human help or fails the agreed phone fallback. Expansion is a separate decision based on reviewed calls, not the absence of complaints.
- Critical scenarios pass repeatedly with unchanged conditions.
- An independent listener completes the same scorecard.
- Known failures become saved tests before the next release.
- Pilot scope, monitoring owner and pause conditions are documented.
- New call reasons or integrations trigger another acceptance review.