DAI-S2S-ST: Evaluating S2S Model Quality with Human Preference Data

Overview

Today we’re releasing DAI-S2S-ST, our single-turn S2S (speech-to-speech) human preference leaderboard, built on 153k comparative human preference ratings across 7 models, 819 distinct prompts, and 12 questions. For this evaluation, we grade model responses to single-turn, pre-recorded input prompts. Our initial results tell a different story than current S2S leaderboards.

We believe the future of voice AI is one where people spend hours a day interacting with models across apps, devices, and new physical interfaces. For that to happen, models need to be more than capable. They need to be compelling enough that people actually want to keep talking to them.

That is different from how voice assistants have traditionally been used. Most were built for short transactions: set a timer, check the weather, play a song. If the model understands the request and completes the task then the interaction is successful, even if the voice sounds robotic or awkward.

Interacting with models for hours a day raises the bar — and requires us to evaluate models differently. Naturalness, personality, empathy, and the overall quality of the interaction become critical. Most voice benchmarks measure whether the model understood the input, answered correctly, or completed the task. These questions matter, but they don’t tell us whether someone would actually want to keep talking to the model. To measure that, we need to ask humans.

DAI-S2S-ST is a first step toward measuring not just what a model can do, but what it feels like to interact with one:

Human preference across 12 dimensions. Existing human preference leaderboards (e.g., Voice Showdown, now retired, and Speech Agent Arena) typically collapse preference into a single overall judgment. DAI-S2S-ST asks twelve distinct side-by-side questions to understand what actually drives preference.
Focused on consumer use cases. We evaluate eleven categories of consumer-oriented scenarios, where the qualities that drive a good interaction differ from what most task-oriented voice-agent leaderboards cover (e.g., τ³-Voice and Eva Bench).

This initial version focuses on single-turn interactions. Future work will extend this evaluation to longer, multi-turn interactions.

What’s In the Evaluation

For full details on our methodology, see the methodology section.

1. Recording

David AI’s evaluation team created 819 single-turn evaluation scripts across eleven objective categories. These scripts span five different difficulty levels and include scene direction across robustness categories (such as background environment).

User-recorded prompts

Objective category

Emotion Response

The speaker shares a personal, emotionally charged situation; the model must recognize the feeling and respond with fitting empathy and tone.

Example promptTranscript
“I keep wondering if there’s something actually wrong with me since I never seem to click with anyone. Is it just in my head?”

Our US-based voice contributor network recorded in a variety of environments that simulate real user conversations.

Indoor Background NoiseCoffee shop
Outdoor Background NoiseOutdoor public park

2. Evaluation

Each recording was run through inference with each of the relevant models, and then David AI’s paid rater panel completed side-by-side preference judgements against our rubric.

These questions are slightly modified from those in our LALM-as-judge vs HITL post based on rater feedback and analysis of those earlier results. Additional details on the evaluation methodology for this leaderboard can be found in the methodology section.

Comparison interface

Input prompt

Model A

Model B

“Overall, which response do you prefer?”

Comparison interface with sample prompt and model outputs. For one pair of responses, raters answer every question in the rubric below.

Every question a rater answers: overall preference, then 5 on Humanness and 6 on Technical Quality.* Only when relevant to input prompt

Results

Current leaderboards do not capture naturalness or acoustic quality

Because of our focus on (1) human perception of conversation quality and (2) consumer-oriented scenarios, our overall rankings diverge from those produced by other public leaderboards with coverage over the same models.

Overall preference by model

Overall preference per model on the CMOS Elo scale, with 95% confidence intervals.

For example, Grok Think Fast 2.0 and Qwen3 both perform very well on leaderboards that measure task completion on transactional voice-agent scenarios with verifiable outcomes, but perform significantly worse on our leaderboard.

Our rubrics resolve into two distinct groups, which we have labeled “Humanness” and “Technical Quality”. We found them by running hierarchical clustering over per-rubric rater responses (after adjusting for rater- and prompt-level effects — see methodology section), which gave a stable clustering at k=2 that consistently reproduced across rater-prompt bootstrapping:

Humanness: conversational register, emotional appropriateness, engagement, naturalness, pleasantness
Technical Quality: instruction following, perceived audio quality, pronunciation accuracy, reasoning, response quality, voice consistency

In previous experiments we grouped questions into “content”, “acoustic” and “interaction” categories. However, during this analysis, we found that these groupings did not capture meaningfully different information — not only did they produce the same model stack ranks, they produced Elo distributions that were statistically indistinguishable from the “overall” human preference Elo.

The root cause of the correlation of these dimensions across models is likely traceable to the model training pipeline: perhaps encoded in the inductive bias of modern architectures, influenced by related latent patterns in the training data, or implicit in the internal metrics different research labs are hill climbing.

These two clusters are a statement about the variation in performance of current models on our prompt distribution. As models improve unevenly across capabilities, or as the use cases evaluated change, these relationships may separate. It is entirely possible that a future model will excel in perceived audio quality and voice consistency but struggle at instruction following and reasoning, just as it is entirely possible that transactional use cases show no strong relationship between human preference and the “humanness” of the model.

Raters prefer certain voices over others

Further evidence for the importance of aesthetics to human preference comes from examining model performance disaggregated by speaking voice. By default, each model is evaluated using two of its supported speaking voices (one male and one female).

Win rate between a model’s two voices

Win rate in direct voice comparison for different models. Tied comparisons are set aside before computing each percentage. Qwen3-Omni is absent because its two voices were never paired against each other in the comparative collection.Brackets show the 95% confidence interval for the leading voice’s share. n.s. — the interval includes an even split, so the lead is not significant at 95%.

Sometimes, as was the case with Grok Ara vs. Rex voices, this preference was very prominent.

Rater comments by Grok voice

Ara

“The models response to the user was accurate as well as consistent and very natural. The overall model response was excellent”

Rex

“model sounds a bit robotic.”
“response needs to sound more energetic.”
“conversation felt a bit unnatural because of models tone of voice.”

Each Grok voice with the free-text comments raters left on it, sampled separately for each — one on Ara, three on Rex, and no comment set against any other; comments appear as raters typed them, spelling and punctuation included.

Users display a strong preference for Ara on Humanness dimensions, and a weaker preference for Ara on Technical Quality dimensions.

Ara’s margin over Rex (Grok voice-think-fast 2.0)

CMOS margin (−3 to +3 scale) and win rate for the two Grok voices, Ara vs. Rex, on Grok voice-think-fast 2.0. Tied comparisons are set aside before computing each win rate. Emotional appropriateness, instruction following and reasoning carry fewer ratings than the rest because those questions are only asked when the prompt calls for them.Bars are the CMOS margin in Ara’s favor, with the 95% confidence interval marked at the bar’s end; 0 is no preference between the two voices. n.s. — the interval includes 0, so the lead is not significant at 95%.

Notably, three Technical Quality dimensions should not vary by voice: reasoning, instruction following, and response quality. In most modern S2S architectures, one model backs every voice, so the underlying quality should be the same. For two models, our data disagrees; Grok and GPT-Realtime both show a voice gap on these dimensions. Raters prefer Grok’s Ara voice on response quality, and its reasoning and instruction-following margins lean the same way. GPT-Realtime shows the same lean between its voices. The reasoning and instruction-following results should be interpreted cautiously; they are gated (applying to only some prompts) and have smaller samples with wide confidence intervals. It’s possible this dynamic is caused by a halo effect from the Humanness preference, or that some models have deeper architectural or training data distribution explanations for the differences in voice performance.

Regardless of the underlying reason, these results demonstrate that you cannot assume comparable performance across model voices as most S2S leaderboards today do. Moreover, they demonstrate that voice-specific factors can strongly influence overall human preference on model outputs.

Higher thinking level does not increase performance on our leaderboard

In contrast to speaking voices, our data showed no material difference in overall model performance based on model thinking level:

Overall Elo by model and thinking level

Highlight rows

Only the three models that ship more than one thinking level appear here — Nova 2 Sonic, Cascade and Qwen3-Omni offer a single setting, so there is nothing to compare. Ordered by Elo within this subset; each plotted range is a 95% confidence interval.

This pattern holds even in isolated dimensions where additional thinking would be most likely to differentiate performance. Higher-thinking variants did not meaningfully out-perform their lower-thinking counterparts in instruction following or reasoning, for example.

We attribute the lack of differentiation among thinking modes primarily to the nature of our task distribution: the single-turn mix does not contain the sort of difficult multi-step problems where you would expect extra thinking to pay off. However, it is notable that the overall differentiation of model thinking levels across S2S leaderboards is mixed — Speech Agent Arena shows relatively little difference between different thinking levels, while τ³-Voice shows a larger effect, but only for certain models. Overall, the public evidence suggests that higher thinking can improve specific reasoning capabilities without reliably improving broader S2S model performance.

Conclusion

Our findings demonstrate that human preference is shaped as much by communication style as by content. This analysis relies on preference data from 283 participants, underscoring the value of human evaluation for measuring outcomes that are difficult to quantify.

At David AI we continue to evaluate public models, pre-release checkpoints, and agentic speech-to-speech workflows using a mixture of human and automated evaluation to better understand real-world interaction quality.

See the leaderboard →

For inquiries regarding model evaluation or future research collaborations, contact evals@withdavid.ai. We are also currently hiring for the Evaluations team.

Bibtex citation:

@misc{davidai2026s2seval,
  title        = {Evaluating Human Preference with David AI’s Single-Turn S2S (DAI-S2S-ST) Human Preference Leaderboard},
  author       = {{David AI Research}},
  year         = {2026},
  howpublished = {\url{https://research.withdavid.ai/}},
  abstract     = {A human-preference leaderboard for speech-to-speech models, built from
                  comparative mean-opinion-score (CMOS) votes by 283 participants. We find
                  that human preference is shaped as much by communication style as by
                  content, underscoring the value of human evaluation for outcomes that are
                  hard to quantify automatically.},
  note         = {Ordinal Bradley-Terry Elo from CMOS preference votes, with two-way
                  prompt-by-rater cluster-bootstrap intervals. Contact: evals@withdavid.ai}
}

Appendix

Extended Data and Robustness Analysis

Evaluation Data

At launch, our leaderboard collection consists of ~153k comparative preference ratings from a pool of 283 raters across model responses to 819 human-recorded audio prompts. A single comparative preference rating is defined as a standard CMOS (-3 to +3 with ties allowed) rating for one of 12 rubric questions for a given pair of audio samples. A rated audio sample pair consists of the output of two model configurations that have been fed the same single-turn human-recorded audio input prompts. A model configuration is defined as the combination of <model, voice, thinking level>, and 19 distinct model configurations were evaluated (generally 2 voices and 2 thinking levels for each model, with a few exceptions in the model selection section). Not all model configurations were compared against all other model configurations for a given input prompt, but every model configuration had at least one comparison against another model configuration for each input prompt.

In addition to CMOS, we also collected MOS ratings for internal validation and comparative analysis with CMOS data. This MOS data is not included in the leaderboard or methodology section, but is discussed throughout the appendix.

The rater panel that supplied these ratings was roughly balanced on gender, with a mean age of 40.9 and coverage across age bands from 18 to 65+. Per collection, 283 raters took part in the CMOS collection and 251 raters in the MOS collection; 166 raters did both MOS and CMOS evaluations.

Inter-rater agreement

In our data, we found a Krippendorff’s alpha for the “Overall” preference question of 0.216, and a Gwet AC2 of 0.211:

However, on average each sample pair has only ~2 independent ratings, making traditional inter-rater agreement metrics difficult to interpret. Instead, we prefer split-half Spearman correlation of overall model ranking, which measures how similar the model rankings are when computed from two disjoint random halves of the rater pool. The table below shows the mean correlation of split-half rankings over 20,000 random splits:

Together, these metrics tell us that per-rating agreement is highly variable across raters, but in aggregate the overall model ranking produced is very stable. More discussion of this phenomenon can be found in the appendix section of our previous blog post.

MOS vs. CMOS Agreement

Our published leaderboard relies on CMOS ratings data to drive Elo scores, but we also collected human MOS ratings on the same set of rubrics on a distinct set of input prompts.

While these two metrics largely agree, there are two notable differences:

MOS flips the Gemini 3.1 Flash Live and GPT-Realtime-2.0/2.1 rankings at the top.
MOS considered the cascade baseline to be competitive with Grok and Nova 2 Sonic, while CMOS considered the cascade baseline to lag far behind.

There are several plausible explanations for why you might see a difference between MOS and CMOS ratings that are worth examining.

Case 1: MOS penalizes catastrophic failures more harshly in saturated domains

The acoustic quality of modern TTS systems is high enough to saturate MOS ratings - almost everything is “pretty good” on an objective scale, and model response that’s in the lowest quartile of output quality might still get a 4 on the MOS scale. In contrast, CMOS asks about relative preference, and the extreme values of the scale simply indicate a “strong” subjective preference for one output over another. A model might have a perfectly good output, but a competing model might still be strongly preferred by the rater. This means that a catastrophic failure earns -3 on CMOS, but so does a merely-strongly-dispreferred output, so CMOS caps the penalty of catastrophic samples while MOS doesn't.

The implication is that model A could be preferred over model B in general, but also have more catastrophic failures, which would lead to model A having a higher CMOS rating but a lower MOS rating. In the case of Gemini 3.1 Flash Live and GPT-Realtime-2.1, we see the following distributions:

In our data, it appears that this effect partially explains the MOS / CMOS discrepancy, but does not completely resolve the question. It is true that GPT-Realtime-2.1 MOS shows more catastrophic failures (MOS scores of 1 and 2) than Gemini 3.1 Flash Live, but this is part of an overall left-shift across the whole distribution; there are also more 3’s and fewer 4’s and 5’s.

Case 2: Heterogeneity in model performance across the input distribution impacts MOS and CMOS asymmetrically

If one model performs characteristically differently on a subset of input prompts, it could have a similar distributional effect to the catastrophic failure case discussed above; model A might be significantly worse at a subset of the eval, but marginally better at the majority, which would lead to stronger CMOS rankings and weaker MOS rankings. To explore this in our data, we examine whether MOS score difference is specific to the difficulty of the prompt:

As expected, the overall MOS score is marginally higher for easier prompts and lower for harder prompts, but the gap between models remains relatively consistent across difficulty levels.

Case 3: Certain dimensions of quality impact overall quality score differently than they impact overall preference

For example, if a rater strongly prefers one speaking voice over another, this could result in a high overall MOS ratings for model A while still showing a strong overall preference for model B on overall CMOS preference. To investigate, we examine the relationship between the MOS / CMOS delta between models across rubric dimensions:

Here again, we find the overall MOS ratings favoring Gemini 3.1 Flash Live across most dimensions, while the CMOS ratings favor GPT-Realtime-2.1 across most dimensions. Interestingly, we do see that perceived audio quality favors Gemini for both MOS and CMOS. One interpretation of this might be that in isolation raters tended to penalize GPT-Realtime for audio quality issues, but this factor did not strongly influence their overall preference across model outputs.

Our conclusion is that MOS and CMOS are simply measuring different things; it is entirely plausible that there are some model quality shortcomings that are relatively more obvious in a standalone pointwise setting, and others that are relatively more obvious in a comparative setting.

Rubric clustering and correlation

Each rubric category bundles several sub-metrics which are themselves positively correlated across prompts within a cluster. Humanness is highly coherent: its five delivery/affect sub-questions (naturalness, pleasantness, engagement, emotional appropriateness, conversational register) move together at a mean pairwise correlation of 0.68, with naturalness and pleasantness the tightest pair (0.82).

Technical Quality is more loosely coupled, with a mean correlation of 0.40. One explanation for this is that it spans two distinct sub-groups: the lexical content items — response quality, instruction following and reasoning — track each other closely at 0.82, while the acoustic content items — perceived audio quality, pronunciation accuracy and voice consistency — cohere more modestly at 0.47. These two sub-groups of Technical Quality correlate only weakly with each other (0.28).

Humanness can be interpreted as a single tight group, while Technical Quality is a broader axis holding a strong lexical content block and a looser audio content block.

These two sub-groups are exactly the groupings that emerge for a k=3 clustering of our data, so it is worth saying why we do not separate them out: we do not find this third cluster to reproduce reliably across rater-prompt bootstrapping (see methodology section). We hypothesize that with more evaluation data and more models being evaluated we would see this third grouping emerge reliably, but our data as yet does not justify treating these as separate clusters, so for the time being we continue using the k=2 grouping of Humanness and Technical Quality that robustly reproduces.

Insights from free text analysis

Our collection protocol includes the ability for raters to leave free text comments explaining their ratings, rationale, or any issues they want to flag that are not captured in existing rubrics. We perform clustering analysis around themes from free-text rater comments and display the results in the table below. Theme mention rate is calculated as percentage of the model’s rater comments from the MOS ratings, where (as discussed in previous post) we see more robust and informative free text comments. Note that this analysis shows the relative differences between the way raters tend to characterize the responses given by each model, rather than the absolute number of comments that addressed the particular theme. Our hope is that these qualitative insights are informative to model trainers in identifying areas of strength and weakness of their model, and that they may influence future model training priorities.

Rubric-Level MOS & CMOS Rankings

Below is the full list of CMOS Elo ranking and MOS scores per model per metric with 95% CI; the MOS score is calculated using sample-averaging. CMOS Elo rankings follow the same BT fitting approach discussed in the methodology section, simply using the specific rubric preference question rather than the overall preference question.

Model Configuration Details

OpenAIGPT-Realtime-2.0-xhigh (Marin)

Config (simplified): Model: GPT-Realtime 2.0 Voice: Marin Reasoning: xhigh Audio: PCM16, 24 kHz in / 24 kHz out Turn detection: off — whole clip sent, then one reply Prompt: default ("You are a helpful voice assistant.")
System prompt: You are a helpful voice assistant.
Full parameters: model: "gpt-realtime-2" type: "realtime" output_modalities: ["audio"] audio: { input: { format: { type: "audio/pcm", rate: 24000 }, turn_detection: null }, output: { format: { type: "audio/pcm", rate: 24000 }, voice: "marin" } } reasoning: { effort: "xhigh" } instructions: "You are a helpful voice assistant."

OpenAIGPT-Realtime-2.0-xhigh (Cedar)

Config (simplified): Model: GPT-Realtime 2.0 Voice: Cedar Reasoning: xhigh Audio: PCM16, 24 kHz in / 24 kHz out Turn detection: off — whole clip sent, then one reply Prompt: default ("You are a helpful voice assistant.")
System prompt: You are a helpful voice assistant.
Full parameters: model: "gpt-realtime-2" type: "realtime" output_modalities: ["audio"] audio: { input: { format: { type: "audio/pcm", rate: 24000 }, turn_detection: null }, output: { format: { type: "audio/pcm", rate: 24000 }, voice: "cedar" } } reasoning: { effort: "xhigh" } instructions: "You are a helpful voice assistant."

OpenAIGPT-Realtime-2.1-xhigh (Marin)

Config (simplified): Model: GPT-Realtime 2.1 Voice: Marin Reasoning: xhigh Audio: PCM16, 24 kHz in / 24 kHz out Turn detection: off — whole clip sent, then one reply Prompt: default ("You are a helpful voice assistant.")
System prompt: You are a helpful voice assistant.
Full parameters: model: "gpt-realtime-2.1" type: "realtime" output_modalities: ["audio"] audio: { input: { format: { type: "audio/pcm", rate: 24000 }, turn_detection: null }, output: { format: { type: "audio/pcm", rate: 24000 }, voice: "marin" } } reasoning: { effort: "xhigh" } instructions: "You are a helpful voice assistant."

OpenAIGPT-Realtime-2.1-xhigh (Cedar)

Config (simplified): Model: GPT-Realtime 2.1 Voice: Cedar Reasoning: xhigh Audio: PCM16, 24 kHz in / 24 kHz out Turn detection: off — whole clip sent, then one reply Prompt: default ("You are a helpful voice assistant.")
System prompt: You are a helpful voice assistant.
Full parameters: model: "gpt-realtime-2.1" type: "realtime" output_modalities: ["audio"] audio: { input: { format: { type: "audio/pcm", rate: 24000 }, turn_detection: null }, output: { format: { type: "audio/pcm", rate: 24000 }, voice: "cedar" } } reasoning: { effort: "xhigh" } instructions: "You are a helpful voice assistant."

OpenAIGPT-Realtime-2.1-minimal (Marin)

Config (simplified): Model: GPT-Realtime 2.1 Voice: Marin Reasoning: minimal Audio: PCM16, 24 kHz in / 24 kHz out Turn detection: off — whole clip sent, then one reply Prompt: default ("You are a helpful voice assistant.")
System prompt: You are a helpful voice assistant.
Full parameters: model: "gpt-realtime-2.1" type: "realtime" output_modalities: ["audio"] audio: { input: { format: { type: "audio/pcm", rate: 24000 }, turn_detection: null }, output: { format: { type: "audio/pcm", rate: 24000 }, voice: "marin" } } reasoning: { effort: "minimal" } instructions: "You are a helpful voice assistant."

OpenAIGPT-Realtime-2.1-minimal (Cedar)

Config (simplified): Model: GPT-Realtime 2.1 Voice: Cedar Reasoning: minimal Audio: PCM16, 24 kHz in / 24 kHz out Turn detection: off — whole clip sent, then one reply Prompt: default ("You are a helpful voice assistant.")
System prompt: You are a helpful voice assistant.
Full parameters: model: "gpt-realtime-2.1" type: "realtime" output_modalities: ["audio"] audio: { input: { format: { type: "audio/pcm", rate: 24000 }, turn_detection: null }, output: { format: { type: "audio/pcm", rate: 24000 }, voice: "cedar" } } reasoning: { effort: "minimal" } instructions: "You are a helpful voice assistant."

xAIGrok Voice Think Fast 2.0 - High (Ara)

Config (simplified): Model: Grok Voice Think Fast 2.0 Voice: Ara Reasoning: high Audio: PCM16, 24 kHz in / 24 kHz out Turn detection: off — whole clip sent, then one reply Prompt: default ("You are a helpful voice assistant.")
System prompt: You are a helpful voice assistant.
Full parameters: model: "grok-voice-think-fast-2.0" session: { instructions: "You are a helpful voice assistant.", voice: "ara", reasoning: { effort: "high" }, turn_detection: { type: null }, audio: { input: { format: { type: "audio/pcm", rate: 24000 } }, output: { format: { type: "audio/pcm", rate: 24000 } } } } Supported voices: ara, eve, leo, rex, sal

xAIGrok Voice Think Fast 2.0 - High (Rex)

Config (simplified): Model: Grok Voice Think Fast 2.0 Voice: Rex Reasoning: high Audio: PCM16, 24 kHz in / 24 kHz out Turn detection: off — whole clip sent, then one reply Prompt: default ("You are a helpful voice assistant.")
System prompt: You are a helpful voice assistant.
Full parameters: model: "grok-voice-think-fast-2.0" session: { instructions: "You are a helpful voice assistant.", voice: "rex", reasoning: { effort: "high" }, turn_detection: { type: null }, audio: { input: { format: { type: "audio/pcm", rate: 24000 } }, output: { format: { type: "audio/pcm", rate: 24000 } } } } Supported voices: ara, eve, leo, rex, sal

xAIGrok Voice Think Fast 2.0 - None (Ara)

Config (simplified): Model: Grok Voice Think Fast 2.0 Voice: Ara Reasoning: none (xAI's own default is high, so this is sent explicitly) Audio: PCM16, 24 kHz in / 24 kHz out Turn detection: off — whole clip sent, then one reply Prompt: default ("You are a helpful voice assistant.")
System prompt: You are a helpful voice assistant.
Full parameters: model: "grok-voice-think-fast-2.0" session: { instructions: "You are a helpful voice assistant.", voice: "ara", reasoning: { effort: "none" }, turn_detection: { type: null }, audio: { input: { format: { type: "audio/pcm", rate: 24000 } }, output: { format: { type: "audio/pcm", rate: 24000 } } } } Supported voices: ara, eve, leo, rex, sal

xAIGrok Voice Think Fast 2.0 - None (Rex)

Config (simplified): Model: Grok Voice Think Fast 2.0 Voice: Rex Reasoning: none (xAI's own default is high, so this is sent explicitly) Audio: PCM16, 24 kHz in / 24 kHz out Turn detection: off — whole clip sent, then one reply Prompt: default ("You are a helpful voice assistant.")
System prompt: You are a helpful voice assistant.
Full parameters: model: "grok-voice-think-fast-2.0" session: { instructions: "You are a helpful voice assistant.", voice: "rex", reasoning: { effort: "none" }, turn_detection: { type: null }, audio: { input: { format: { type: "audio/pcm", rate: 24000 } }, output: { format: { type: "audio/pcm", rate: 24000 } } } } Supported voices: ara, eve, leo, rex, sal

GoogleGemini 3.1 Flash Live Preview - thinking-high (Zephyr)

Config (simplified): Model: Gemini 3.1 Flash Live Preview Voice: Zephyr Thinking: high — thoughts not returned Audio: PCM16, 16 kHz in / 24 kHz out Turn detection: off — automatic activity detection disabled Extras: output audio transcription on Prompt: default ("You are a helpful voice assistant.")
System prompt: You are a helpful voice assistant.
Full parameters: model: "gemini-3.1-flash-live-preview" responseModalities: ["AUDIO"] systemInstruction: "You are a helpful voice assistant." speechConfig: { voiceConfig: { prebuiltVoiceConfig: { voiceName: "Zephyr" } } } realtimeInputConfig: { automaticActivityDetection: { disabled: true } } thinkingConfig: { thinkingLevel: "HIGH", includeThoughts: false } outputAudioTranscription: {} realtimeInput: { audio: { data: <b64>, mimeType: "audio/pcm;rate=16000" } } Output audio: 24000 Hz PCM16

GoogleGemini 3.1 Flash Live Preview - thinking-high (Iapetus)

Config (simplified): Model: Gemini 3.1 Flash Live Preview Voice: Iapetus Thinking: high — thoughts not returned Audio: PCM16, 16 kHz in / 24 kHz out Turn detection: off — automatic activity detection disabled Extras: output audio transcription on Prompt: default ("You are a helpful voice assistant.")
System prompt: You are a helpful voice assistant.
Full parameters: model: "gemini-3.1-flash-live-preview" responseModalities: ["AUDIO"] systemInstruction: "You are a helpful voice assistant." speechConfig: { voiceConfig: { prebuiltVoiceConfig: { voiceName: "Iapetus" } } } realtimeInputConfig: { automaticActivityDetection: { disabled: true } } thinkingConfig: { thinkingLevel: "HIGH", includeThoughts: false } outputAudioTranscription: {} realtimeInput: { audio: { data: <b64>, mimeType: "audio/pcm;rate=16000" } } Output audio: 24000 Hz PCM16

GoogleGemini 3.1 Flash Live Preview - thinking-minimal (Zephyr)

Config (simplified): Model: Gemini 3.1 Flash Live Preview Voice: Zephyr Thinking: minimal — thoughts not returned Audio: PCM16, 16 kHz in / 24 kHz out Turn detection: off — automatic activity detection disabled Extras: output audio transcription on Prompt: default ("You are a helpful voice assistant.")
System prompt: You are a helpful voice assistant.
Full parameters: model: "gemini-3.1-flash-live-preview" responseModalities: ["AUDIO"] systemInstruction: "You are a helpful voice assistant." speechConfig: { voiceConfig: { prebuiltVoiceConfig: { voiceName: "Zephyr" } } } realtimeInputConfig: { automaticActivityDetection: { disabled: true } } thinkingConfig: { thinkingLevel: "MINIMAL", includeThoughts: false } outputAudioTranscription: {} realtimeInput: { audio: { data: <b64>, mimeType: "audio/pcm;rate=16000" } } Output audio: 24000 Hz PCM16

GoogleGemini 3.1 Flash Live Preview - thinking-minimal (Iapetus)

Config (simplified): Model: Gemini 3.1 Flash Live Preview Voice: Iapetus Thinking: minimal — thoughts not returned Audio: PCM16, 16 kHz in / 24 kHz out Turn detection: off — automatic activity detection disabled Extras: output audio transcription on Prompt: default ("You are a helpful voice assistant.")
System prompt: You are a helpful voice assistant.
Full parameters: model: "gemini-3.1-flash-live-preview" responseModalities: ["AUDIO"] systemInstruction: "You are a helpful voice assistant." speechConfig: { voiceConfig: { prebuiltVoiceConfig: { voiceName: "Iapetus" } } } realtimeInputConfig: { automaticActivityDetection: { disabled: true } } thinkingConfig: { thinkingLevel: "MINIMAL", includeThoughts: false } outputAudioTranscription: {} realtimeInput: { audio: { data: <b64>, mimeType: "audio/pcm;rate=16000" } } Output audio: 24000 Hz PCM16

AmazonNova 2 Sonic (Tiffany)

Config (simplified): Model: Nova 2 Sonic (Bedrock, us-east-1) Voice: Tiffany Reasoning: n/a — no effort control Audio: LPCM16 mono, 16 kHz in / 24 kHz out Turn detection: cannot be disabled — endpointing LOW (~2.0 s pause), the most patient setting Sampling: model defaults (no max tokens, top-p or temperature set) Prompt: default ("You are a helpful voice assistant.")
System prompt: You are a helpful voice assistant.
Full parameters: model: "amazon.nova-2-sonic-v1:0" (bedrock, us-east-1) sessionStart.inferenceConfiguration: {} # maxTokens / topP / temperature all unset sessionStart.turnDetectionConfiguration: { endpointingSensitivity: "LOW" } promptStart.textOutputConfiguration: { mediaType: "text/plain" } promptStart.audioOutputConfiguration: { mediaType: "audio/lpcm", sampleRateHertz: 24000, sampleSizeBits: 16, channelCount: 1, audioType: "SPEECH", encoding: "base64", voiceId: "tiffany" } contentStart(SYSTEM,TEXT) + textInput.content: "You are a helpful voice assistant." contentStart(USER,AUDIO).audioInputConfiguration: { mediaType: "audio/lpcm", sampleRateHertz: 16000, sampleSizeBits: 16, channelCount: 1, audioType: "SPEECH", encoding: "base64" }

AmazonNova 2 Sonic (Matthew)

Config (simplified): Model: Nova 2 Sonic (Bedrock, us-east-1) Voice: Matthew Reasoning: n/a — no effort control Audio: LPCM16 mono, 16 kHz in / 24 kHz out Turn detection: cannot be disabled — endpointing LOW (~2.0 s pause), the most patient setting Sampling: model defaults (no max tokens, top-p or temperature set) Prompt: default ("You are a helpful voice assistant.")
System prompt: You are a helpful voice assistant.
Full parameters: model: "amazon.nova-2-sonic-v1:0" (bedrock, us-east-1) sessionStart.inferenceConfiguration: {} # maxTokens / topP / temperature all unset sessionStart.turnDetectionConfiguration: { endpointingSensitivity: "LOW" } promptStart.textOutputConfiguration: { mediaType: "text/plain" } promptStart.audioOutputConfiguration: { mediaType: "audio/lpcm", sampleRateHertz: 24000, sampleSizeBits: 16, channelCount: 1, audioType: "SPEECH", encoding: "base64", voiceId: "matthew" } contentStart(SYSTEM,TEXT) + textInput.content: "You are a helpful voice assistant." contentStart(USER,AUDIO).audioInputConfiguration: { mediaType: "audio/lpcm", sampleRateHertz: 16000, sampleSizeBits: 16, channelCount: 1, audioType: "SPEECH", encoding: "base64" }

Alibaba (self-hosted, Modal)Qwen3-Omni-30B-A3B-Instruct (Chelsie)

Config (simplified): Model: Qwen3-Omni 30B-A3B Instruct — self-hosted on Modal (H200) Voice: Chelsie Reasoning: n/a Audio: WAV PCM16, 16 kHz in / 24 kHz out Turn detection: n/a — one clip per call, batched Prompt: Qwen's own official assistant prompt. Caps replies at 50 words, forbids formatting.
System prompt: You are a virtual voice assistant with no gender or age. You are communicating with the user. In user messages, "I/me/my/we/our" refer to the user and "you/your" refer to the assistant. In your replies, address the user as "you/your" and yourself as "I/me/my"; never mirror the user's pronouns—always shift perspective. Keep original pronouns only in direct quotes; if a reference is unclear, ask a brief clarifying question. Interact with users using short (no more than 50 words), brief, straightforward language, maintaining a natural tone. Never use formal phrasing, mechanical expressions, bullet points, overly structured language. Your output must consist only of the spoken content you want the user to hear. Do not include any descriptions of actions, emotions, sounds, or voice changes. Do not use asterisks, brackets, parentheses, or any other symbols to indicate tone or actions. You must answer users' audio or text questions, do not directly describe the video content. You should communicate in the same language strictly as the user unless they request otherwise. When you are uncertain (e.g., you can't see/hear clearly, don't understand, or the user makes a comment rather than asking a question), use appropriate questions to guide the user to continue the conversation. Keep replies concise and conversational, as if talking face-to-face
Full parameters: model: "Qwen/Qwen3-Omni-30B-A3B-Instruct" (self-hosted) endpoint: modal://main/qwen3-omni-instruct/Qwen3OmniS2SBatched.generate task: "s2s" speaker: "chelsie" messages: [ { role: "system", content: [{ type: "text", text: <official prompt> }] }, { role: "user", content: [{ type: "audio", audio_b64: <WAV b64> }] } ] Input audio: WAV PCM16 16000 Hz Output audio: 24000 Hz PCM16

Alibaba (self-hosted, Modal)Qwen3-Omni-30B-A3B-Instruct (Ethan)

Config (simplified): Model: Qwen3-Omni 30B-A3B Instruct — self-hosted on Modal (H200) Voice: Ethan Reasoning: n/a Audio: WAV PCM16, 16 kHz in / 24 kHz out Turn detection: n/a — one clip per call, batched Prompt: Qwen's own official assistant prompt. Caps replies at 50 words, forbids formatting.
System prompt: You are a virtual voice assistant with no gender or age. You are communicating with the user. In user messages, "I/me/my/we/our" refer to the user and "you/your" refer to the assistant. In your replies, address the user as "you/your" and yourself as "I/me/my"; never mirror the user's pronouns—always shift perspective. Keep original pronouns only in direct quotes; if a reference is unclear, ask a brief clarifying question. Interact with users using short (no more than 50 words), brief, straightforward language, maintaining a natural tone. Never use formal phrasing, mechanical expressions, bullet points, overly structured language. Your output must consist only of the spoken content you want the user to hear. Do not include any descriptions of actions, emotions, sounds, or voice changes. Do not use asterisks, brackets, parentheses, or any other symbols to indicate tone or actions. You must answer users' audio or text questions, do not directly describe the video content. You should communicate in the same language strictly as the user unless they request otherwise. When you are uncertain (e.g., you can't see/hear clearly, don't understand, or the user makes a comment rather than asking a question), use appropriate questions to guide the user to continue the conversation. Keep replies concise and conversational, as if talking face-to-face
Full parameters: model: "Qwen/Qwen3-Omni-30B-A3B-Instruct" (self-hosted) endpoint: modal://main/qwen3-omni-instruct/Qwen3OmniS2SBatched.generate task: "s2s" speaker: "ethan" messages: [ { role: "system", content: [{ type: "text", text: <official prompt> }] }, { role: "user", content: [{ type: "audio", audio_b64: <WAV b64> }] } ] Input audio: WAV PCM16 16000 Hz Output audio: 24000 Hz PCM16

ElevenLabs, OpenAI, ElevenLabsCascade

Config (simplified): Pipeline: ElevenLabs Scribe v2 (ASR) -> OpenAI GPT-5.6-terra (LLM) -> ElevenLabs Flash v2.5 (TTS) Voice: Rachel (ElevenLabs legacy default, retires 2026-12-31) Reasoning: low — on the LLM leg Audio: PCM, 16 kHz in / 24 kHz out Turn detection: n/a — one clip per call Language: auto (not pinned) Prompt: pinned by the deployed app, cannot be overridden. Caps replies at 100 words, forbids markdown.
System prompt: You are a helpful assistant handling a voice chat with a user. # Important Voice Considerations 1. Respond naturally and conversationally as you would in a real conversation 2. Try to be helpful and always follow the instructions below. 3. Keep the response short and concise. Do not exceed 100 words. You are writing a final script for text-to-speech (TTS). Your response will be synthesized directly into speech. Follow the duration instruction as strictly as possible. Output only the final spoken text, with natural punctuation. Do not output markdown, bullets, JSON, XML tags, stage directions, or extra commentary. Do not mention these instructions.
Full parameters: endpoint: modal://main/s2s-cascaded/SingleTurnCascade.run_turn asr_model: "scribe_v2" (ElevenLabs Scribe v2) llm_model: "gpt-5.6-terra" (OpenAI) tts_provider: "elevenlabs" tts_model: "eleven_flash_v2_5" voice: "21m00Tcm4TlvDq8ikWAM" (Rachel; legacy default, retires 2026-12-31) language: null reasoning_effort: "low" messages: [{ role: "user", content: [{ type: "audio", audio_b64: <WAV b64>, filename: "prompt.wav" }] }] Input audio: 16000 Hz Output audio: 24000 Hz PCM No system-prompt input: the deployed app pins its own and 400s on a system turn.

References

Bradley, R. A., & Terry, M. E. (1952). Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39(3/4), 324–345.

Chiang, W.-L., et al. (2024). Chatbot Arena: an open platform for evaluating LLMs by human preference.

Elo, A. E. (1978). The Rating of Chessplayers, Past and Present. Arco.

Eyben, F., Scherer, K. R., Schuller, B. W., et al. (2016). The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for Voice Research and Affective Computing. IEEE Transactions on Affective Computing, 7(2), 190–202.

Krippendorff, K. (2004). Content Analysis: An Introduction to Its Methodology (2nd ed.). Sage.

Gwet, K. L. (2008). Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1), 29–48.

McCullagh, P. (1980). Regression models for ordinal data. JRSS: Series B, 42(2), 109–142. (cumulative-link / ordinal model)

Massey, K. (1997). Statistical models applied to the rating of sports teams. Bluefield College.

Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428.