Voice Agent Simulation Bench for utility billing and payment assistance . Every call chains two to four caller requests, over street, café or television noise, to a calm or angry caller. Every world effect and spoken figure is graded against the trace.
Leaderboard
Voice agent Task completion Latency Turns Barge-in/call Cost Type Model
grok voice 44.7% 2.64s 16 1.07 ~$0.210 speech‑to‑speech Grok Voice Think Fast 2.0 gemini 3.8 live 40.1% 1.66s 12 0.13 ~$0.193 speech‑to‑speech Gemini 3.8 Live gpt-live 1 39.2% 1.62s 17 1.05 ~$0.192 speech‑to‑speech GPT-Live 1 openai realtime 2.1 37.9% 2.12s 14 0.44 ~$0.633 speech‑to‑speech GPT Realtime 2.1 elevenlabs 36.4% 2.18s 12 0.04 ~$0.272 cascade ElevenLabs ASR → gpt-4.1-mini → eleven_flash_v2 openai realtime 33.6% 2.32s 14 0.35 ~$0.607 speech‑to‑speech GPT Realtime 2 livekit 29.4% 2.96s 12 0.04 ~$0.077 cascade Deepgram nova-3 → gpt-4.1-mini → ElevenLabs eleven_flash_v2 gemini 3.1 live 25.6% 2.66s 12 0.10 ~$0.199 speech‑to‑speech Gemini 3.1 Flash Live mistral 22.7% 3.75s 12 0.08 ~$0.279 cascade Voxtral realtime → mistral-medium-2604 → Voxtral TTS gradium 21.3% 2.42s 14 1.18 ~$0.120 cascade Gradium STT → gpt-4.1-mini → Gradium TTS vapi 20.5% 2.08s 20 0.71 ~$0.271 cascade Deepgram → gpt-4.1-mini → ElevenLabs deepgram 19.8% 2.04s 13 0.34 ~$0.193 cascade nova-3 → gpt-4.1-mini → aura-2 hugging face 19.0% 5.28s 12 9.86 ~$0.053 cascade Whisper large-v3 → gpt-oss-120b → Kokoro-82M pipecat 17.3% 1.96s 18 7.94 ~$0.109 cascade Deepgram nova-3 → gpt-4.1-mini → ElevenLabs eleven_flash_v2
Cost vs. completed tasks
10%
20%
30%
40%
50%
$0.05
$0.10
$0.20
$0.50
Mean cost per call (USD, log scale) · Y: task completion (% of calls)
Pareto frontier
grok voice: 44.7% completion, ~$0.210 per call grok voice: 44.7% completion, ~$0.210 per call gemini 3.8 live: 40.1% completion, ~$0.193 per call 3.8 Live gemini 3.8 live: 40.1% completion, ~$0.193 per call gpt-live 1: 39.2% completion, ~$0.192 per call GPT-Live gpt-live 1: 39.2% completion, ~$0.192 per call openai realtime 2.1: 37.9% completion, ~$0.633 per call Realtime 2.1 openai realtime 2.1: 37.9% completion, ~$0.633 per call elevenlabs: 36.4% completion, ~$0.272 per call elevenlabs: 36.4% completion, ~$0.272 per call openai realtime: 33.6% completion, ~$0.607 per call Realtime 2 openai realtime: 33.6% completion, ~$0.607 per call livekit: 29.4% completion, ~$0.077 per call livekit: 29.4% completion, ~$0.077 per call gemini 3.1 live: 25.6% completion, ~$0.199 per call 3.1 Live gemini 3.1 live: 25.6% completion, ~$0.199 per call mistral: 22.7% completion, ~$0.279 per call mistral: 22.7% completion, ~$0.279 per call gradium: 21.3% completion, ~$0.120 per call gradium: 21.3% completion, ~$0.120 per call vapi: 20.5% completion, ~$0.271 per call vapi: 20.5% completion, ~$0.271 per call deepgram: 19.8% completion, ~$0.193 per call deepgram: 19.8% completion, ~$0.193 per call hugging face: 19.0% completion, ~$0.053 per call hugging face: 19.0% completion, ~$0.053 per call pipecat: 17.3% completion, ~$0.109 per call pipecat: 17.3% completion, ~$0.109 per call grok voice: 44.7% completion, ~$0.210 per call grok voice: 44.7% completion, ~$0.210 per call grok voicegemini 3.8 live: 40.1% completion, ~$0.193 per call gemini 3.8 live: 40.1% completion, ~$0.193 per call gemini 3.8 livegpt-live 1: 39.2% completion, ~$0.192 per call gpt-live 1: 39.2% completion, ~$0.192 per call gpt-live 1openai realtime 2.1: 37.9% completion, ~$0.633 per call openai realtime 2.1: 37.9% completion, ~$0.633 per call openai realtime 2.1elevenlabs: 36.4% completion, ~$0.272 per call elevenlabs: 36.4% completion, ~$0.272 per call elevenlabsopenai realtime: 33.6% completion, ~$0.607 per call openai realtime: 33.6% completion, ~$0.607 per call openai realtimelivekit: 29.4% completion, ~$0.077 per call livekit: 29.4% completion, ~$0.077 per call livekitgemini 3.1 live: 25.6% completion, ~$0.199 per call gemini 3.1 live: 25.6% completion, ~$0.199 per call gemini 3.1 livemistral: 22.7% completion, ~$0.279 per call mistral: 22.7% completion, ~$0.279 per call mistralgradium: 21.3% completion, ~$0.120 per call gradium: 21.3% completion, ~$0.120 per call gradiumvapi: 20.5% completion, ~$0.271 per call vapi: 20.5% completion, ~$0.271 per call vapideepgram: 19.8% completion, ~$0.193 per call deepgram: 19.8% completion, ~$0.193 per call deepgramhugging face: 19.0% completion, ~$0.053 per call hugging face: 19.0% completion, ~$0.053 per call hugging facepipecat: 17.3% completion, ~$0.109 per call pipecat: 17.3% completion, ~$0.109 per call pipecat
Build this around your own systems
This board is one exam, on one utility’s policies. Veris builds the same thing around yours: your workflows, your systems, your rules, and the calls your agents actually get. Simulated callers, twinned services, and a verdict from the world the agent leaves behind rather than the transcript it produced.
Book a demo
Running a voice stack of your own? We’ll put it through the same 100 tasks and publish the result. Contact us at hello@veris.ai