Everyone benchmarks the model. We benchmark the stack around it.
An independent benchmark isolating the orchestration layer: turn-taking, interruptions, latency and tool execution. One byte-identical agent, run across 6 platforms. The differences are the platforms, not the model. Note: these are platform defaults, expect higher peaks once you tune to your use case.
Figures are comparisons under one fixed harness · see methodology.
Highest reliability
96.6%
pass^3
Fastest responses
1.73s
median turn latency
Best interruption handling
4.90
out of 5 · interruption score
Most consistent latency
1.66s-2.95s
P5-P95 turn latency
By metric
Per-turn latency · line = median (P50), box P25–P75, whiskers P5–P95 · ~1,100–1,570 turns/platform
Latency
With the model, prompt and TTS fixed, per-turn latency reflects the platform’s turn-taking: endpointing, VAD, buffering and network path. ElevenLabs is fastest (median 1.73 s per turn)
but carries a long tail (P95 3.19 s, max 10.4 s). Vapi is the most consistent (P5–P95 1.66–2.95 s). LiveKit now sits near Vapi at 2.46 s median; Pipecat and Synthflow remain slower, around 3.15 s.
Average interruption score (0–5) · interruption scenarios only
Interruption
Turn-taking is set by the platform, not the prompt: Voice Activity Detection and end-of-turn detection decide when the agent yields. Across the interruption scenarios (barge-in, cough, mid-turn silence),
Pipecat scores highest at 4.90, with LiveKit close at 4.89. Vapi is lowest at 4.63, and the field is tight overall.
How Interruption Score is determined⌄
Cekura uses voice-activity detection on the stereo recording to find every point where the Main Agent starts speaking while the caller is still talking, then scores 5 × (1 − interruptions ÷ turns), clamped to 0–5, so 5/5 means it never talked over the caller. We report it on the interruption scenarios (those that inject barge-in, coughs, or mid-turn silence) because a whole-suite average is diluted by calls with nothing to interrupt.
Appropriate end-call rate (% of calls)
Call handling
Whether a call ends cleanly is handled by the platform, not the model.
Retell ended every call correctly; Vapi, Pipecat and LiveKit were next at 99.4%. Synthflow was lowest at 94.9%, usually failing to close after an emergency redirect or a late second request. Native end-call was disabled; ending was specified in the prompt.
How Appropriate End Call is determined⌄
A per-call pass/fail from an LLM judge that reads the transcript and the reason the call ended, and decides whether the Main Agent wrapped up properly: resolving the request before hanging up, and not looping past a clean goodbye. Native end-call was disabled for the benchmark, so this reflects how each platform executes a prompt-driven hangup rather than a built-in one.
Average repetition score (0–5)
Conversation
Loops surface when a platform mishandles silence or overlapping speech: a turn-taking failure, not a prompt one.
Scores run 4.61 to 4.74, Retell highest. The metric sits near its ceiling, so read the low tail, not the mean.
How Repetition Score is determined⌄
An LLM flags each time the Main Agent re-confirms the same, unchanged information twice or more; the score is 5 × (1 − repetitions ÷ turns), clamped to 0–5, so 5/5 means no unnecessary repetition. Most calls sit near the ceiling; the signal is the low tail, where a platform loops instead of recovering with one clean re-ask or a clean close.
Average voice-tone score (0–5)
Experience
Audio quality is set by the platform’s media pipeline (encoding, jitter, packet handling), not the model.
ElevenLabs is highest at 4.60/5; Retell, Synthflow and Pipecat cluster from 4.45 to 4.51, while Vapi and the newer LiveKit run land at 4.20 and 4.04.
How Voice Tone is determined⌄
A dedicated ML model analyzes the Main Agent's audio channel, not the transcript, scoring clarity (how clear and understandable the voice is) and jitter (timing variation that degrades audio). It catches calls that complete the task but still sound noisy, clipped, or degraded.
Full data
| Platform | pass^1 | pass^3 | Lat P50 | Lat P95 | Interrupt | End call | Repetition | Voice tone |
|---|---|---|---|---|---|---|---|---|
| Retell | 98.9% | 96.6% | 1.96s | 3.79s | 4.79 | 100.0% | 4.74 | 4.51 |
| Vapi | 98.3% | 94.9% | 2.34s | 2.95s | 4.63 | 99.4% | 4.66 | 4.20 |
| Pipecat | 95.5% | 89.8% | 3.15s | 5.42s | 4.90 | 98.3% | 4.61 | 4.45 |
| LiveKit | 94.9% | 84.7% | 2.46s | 3.87s | 4.89 | 99.4% | 4.64 | 4.04 |
| Synthflow | 90.4% | 81.4% | 3.16s | 5.08s | 4.66 | 94.9% | 4.69 | 4.47 |
| ElevenLabs | 88.1% | 76.3% | 1.73s | 3.19s | 4.77 | 96.6% | 4.61 | 4.60 |
pass^1 = one run meets the success rubric; pass^3 = all three runs do; the gap is the consistency signal. Latency is Main-Agent per-turn (ms). Interruption, repetition and voice-tone are on a 0–5 scale (higher is better).
The suite separates providers most in Workflow Complexity & Recovery (tool-failure, multi-step, recovery), a 33.3-point spread, and least in red-team safety and voice robustness.
| Category | Retell | Vapi | Pipecat | LiveKit | Synthflow | ElevenLabs |
|---|---|---|---|---|---|---|
| Positive / Core Scheduling | 87.5% | 100.0% | 100.0% | 75.0% | 100.0% | 87.5% |
| Workflow Complexity & Recovery | 95.2% | 85.7% | 95.2% | 76.2% | 76.2% | 61.9% |
| Voice Robustness & Turn-Taking | 100.0% | 100.0% | 80.0% | 92.0% | 80.0% | 80.0% |
| Red Team, Safety & Privacy | 100.0% | 100.0% | 100.0% | 100.0% | 80.0% | 100.0% |
One agent, deployed unchanged on six platforms, called over the phone by the same scenarios and scored by the same evaluators — so what varies between platforms is the layer the numbers measure.
Every platform runs the same agent: Ava, the scheduling assistant for a fictional healthcare practice, Cedarwood Family Clinic. Ava books, reschedules, cancels and looks up appointments over the phone, backed by four tools (lookup_patient, check_availability, book_appointment and cancel_appointment) against fixed mock data. Its prompt carries clinic guardrails: no medical advice, 911 for emergencies, minimal PII, and no disclosure of its own instructions.
The orchestration layer: turn-taking, interruption and barge-in, latency, tool execution and call handling, isolated by holding the agent constant. Components (STT/TTS) and model reasoning are measured elsewhere; this isolates the orchestration between them.
The agent is held constant wherever each platform allows it: a byte-identical system prompt (SHA-verified) and first message, the same four tool definitions and mock data, and the same pinned LLM (gpt-4.1, temp 0), STT, TTS and voice. What varies is each platform’s own SDK integration and orchestration stack: the layer under comparison. Where a platform can’t accept a pin, it is documented in Stack parity, below.
59 evaluators across four categories (Positive / Core Scheduling, Workflow Complexity & Recovery, Voice Robustness & Turn-Taking, and Red Team, Safety & Privacy), each run 3 times. Every scenario targets a single expected outcome.
Each pass is determined by a rubric configured for the scenario. For scenarios where a correct end state determines success, mock tool accuracy is used as a proxy for that state, alongside the order in which the tools are called. In red-teaming scenarios, success is determined by expected outcome, graded by an LLM judge. Human-reviewed successes count as effective passes for their configured rubric gates; raw metric scores are retained separately. See examples.
The agent is aligned across all six platforms; the remaining differences are documented platform constraints, not prompt drift.
| Layer | Pinned to | Parity across platforms |
|---|---|---|
| LLM | gpt-4.1 · temp 0 | Pinned across all six. Managed legs use platform model pools; LiveKit via its Inference gateway; our Pipecat leg uses an Azure OpenAI key. |
| STT | Deepgram nova-3 | Pinned on Vapi, Synthflow, LiveKit and Pipecat. Retell exposes only a coarse mode; ElevenLabs forces Scribe. |
| TTS | ElevenLabs eleven_flash_v2 | Pinned where the platform exposes it. |
| Voice | ElevenLabs Brian | Exact voice ID on all except Retell, which uses a name-proxy. |
| Prompt & tools | Byte-identical | Same system prompt (SHA-verified), first message, four tool definitions and mock data on every platform. |
| Telephony | PSTN / SIP | All scored runs are over the phone; native vs. SIP-bridged path varies by platform. |
Same evaluators everywhere
Every platform is scored by the same evaluators, applied identically, three runs each.
Managed vs. framework
Vapi, Retell and ElevenLabs are managed platforms that run the agent for us. LiveKit and Pipecat are frameworks: they provide the real-time orchestration under test (the session, SIP/PSTN media, VAD, end-of-turn and turn-taking) while we host the agent worker that wires in the same prompt, tool contract and core stack (gpt-4.1, Deepgram nova-3, ElevenLabs Flash) as the managed providers.
Where the workers ran
For LiveKit the agent was deployed on LiveKit Cloud, with LiveKit infrastructure serving all three models (STT, LLM and TTS), bridged into the calls over PSTN/SIP; Pipecat ran on Pipecat Cloud (Daily-backed).
End-call handling
Each platform's built-in end-call was disabled; the agent was instructed to end the call from the system prompt, and the testing agent does not end calls. This shapes the End Call and Repetition results.
Orchestration, not the model
The emphasized metrics (latency, interruption, end-call, repetition) are largely determined by Voice Activity Detection, End-of-Turn detection and other orchestration conditions, rather than the model.
Telephony and pins
Telephony path differs by platform (native vs. SIP-bridged); STT and TTS are pinned with per-platform exceptions (see Stack parity).
Cascade only
Only cascade platforms are included in this version; near-duplex / realtime is a separate track.
Note · Framework baselines
LiveKit and Pipecat were built from the providers’ standard, publicly available agent templates. Their results are a baseline for the default implementation path, not the theoretical ceiling of what each framework can achieve. With deeper tuning, custom orchestration logic and production-specific optimization, teams can likely push several of these numbers higher.
01
Every figure is a comparison under one fixed test harness, not a production success rate.
02
Use it to narrow the field to a few candidates, then validate them on your own traffic.
03
High-volume support weights reliability and consistency; premium CX weights latency and turn-taking; regulated workflows weight accuracy.
Full run collections, with transcripts.