Show HN: Zeus – Claude Code for AI voice agents
thunderphone.comHi everyone, Alex here - We’re excited to announce the launch of Zeus, which is a platform that makes AI phone calls actually work, and without requiring extensive hand-tuning!
(The below is a long description even after we made it shorter! If you prefer video, there's one covering the main points here: https://www.youtube.com/watch?v=kIqp37udd6s.)
When we started building business phone agents in 2024 (time flies…) we kept running into technical problems that ruined phone calls, caused by the limitations of existing architectures. The two biggest issues were that phone agents would mishear callers, and that they would disobey scripts or otherwise respond in weird ways to what callers said.
Both were consequences of the prevailing architecture for voice agents - typically, a “three step” or “cascaded” pipeline: one transcription model turning the audio of what the caller said into text, one LLM to generate a response (and sometimes tool calls) based on that text, and finally a TTS model to generate response audio from the response text.
This architecture caused the “mishearing callers” problem because transcription models aren’t perfect. Especially in challenging situations like noisy environments or unclear callers, languages other than English, and short utterances like “yeah”, transcription models frequently make mistakes.
The issue with the pipeline approach is that when the caller’s audio is boiled down to a single “best guess” text transcript, mistakes are often unrecoverable: there’s typically no information on “top n” best guesses or other signal from the audio for the LLM to use if the original transcript was wrong.
The pipeline approach also causes the "weird response" problem because of the tradeoff between LLM speed and intelligence. Phone calls demand fast responses, but this is in tension with LLMs' need to “think”, i.e. run inference often for several seconds, to generate a correct response. The obvious choice is to pick a fast enough model to make a phone call feel responsive - but then you get an unavoidably high error rate.
Once we realized these problems, we decided to build our own stack to address them. Over the past year and a half, we've largely solved them by combining numerous models on each call.
Audio misunderstanding problems are handled in two primary ways. We run multiple, even numerous, transcription models on most calls, giving more “shots on goal” to get the correct transcript from the original audio. Even if none of them get it exactly right, multiple almost-correct answers can give enough information to triangulate on a correct transcript. We then use one or more LLMs to make a best-guess choice as to what the caller actually said. In challenging situations, we even pipe the caller audio direct to one or more audio-input LLMs to further improve listening comprehension.
On the instruction-following side, we take a similar approach. We run a mix of fast and slow LLMs on conversation turns to combine snappy responses with more thorough thinking behind the scenes. The way this looks in practice is a multi-staged classification problem, where after 50ms, after 500ms, after 1s, etc. we’re simultaneously generating responses with more and more inference and also classifying whether we need to “think harder” to generate a definitive correct answer.
In combination, these two approaches have been very effective. Qualitatively, we’ve seen calls from our real customers go much more smoothly for callers and make it to successful outcomes more frequently than when they were running on three-step or speech-to-speech pipelines. Quantitatively, we’ve set a record of 99.4% on the Big Bench Audio benchmark, smarter than any other model that’s fast enough to handle real calls. (We’re also in the process of benchmarking ThunderPhone on more public benchmarks - stay tuned.)
The downside of our approach is that it can be very complex to configure. Every agent benefits from a different setup on the ThunderPhone stack, and even individual call turns on the same call often benefit from running on a different stack. We’ve been able to configure this complexity manually for enterprise customers, but found it prohibitive to give smaller self-serve customers the full benefits of our approach without being able to give their agents our full attention.
That’s how Zeus came about. We realized that the configurations we were doing for enterprise customers were generally repetitive enough that an AI could do them - we just needed to teach the AI how to do so.
We also realized that we needed to give the AI more tools to be effective configuring customers’ agents. We’re big fans of Claude Code and Codex for software development, but they fall short on configuring voice agents for a few reasons.
One is that unlike software development, where it’s possible to compile code and run tests to see if the code actually works, it’s not possible to run live calls within a computer right after configuring a voice agent - and coding agents don’t even know how to run simulated calls out of the box, or know how to “listen” to a call to spot real problems. We built both capabilities into Zeus, so that it’s able to have a “closed loop” experience building, testing, and improving voice agents just like coding agents are able to with regular software.
Another issue is that coding agents are gullible when it comes to marketing claims! Companies building AI infrastructure, including voice AI, make the most favorable claims about their capabilities, and post them widely online. Claude and Codex read all that and tend to believe it uncritically - and then make tool decisions based on marketing rather than real performance. At best, that leads to suboptimal architectural choices; at worst, it leads to voice agents that don’t really work at all.
We’ve sidestepped this problem with a system of continuous evals feeding an internal knowledge base. For all aspects of call performance that make a difference for voice agents - accurate audio understanding and instruction following, but also other things like noise cancelling, turn-taking, and more - we have internal benchmarks based on data from real calls that we use to test the performance of AI models. Every time a new AI model comes out, we (now mostly automatically) feed it through our evals to see how it does on performance, speed, cost, and reliability/consistency. This then feeds our internal knowledge base, which Zeus uses to pick the right stack for every agent and every call and conversation turn.
This all happens behind the scenes - our aim has been to make the experience for a ThunderPhone user as seamless as possible. In our opinion, all a ThunderPhone user should have to do is describe their use case and goals, and Zeus helps fill in any ambiguities, connect VoIP and software integrations, and then optimize the voice agent(s) in question on simulated and live calls until they’re getting the highest performance on the user’s business goals (conversions, unit economics, etc.) possible based on AI models available today.
Zeus is also differentiated based on cost. Zeus itself is free to use for now, though we expect to move to subscription or token-based pricing in the future. But for calls themselves, we’ve become the first voice AI vendor to our knowledge to charge on a cost-plus basis. That means that for every call, ThunderPhone customers pay exactly what we pay our vendors, plus a small upcharge: we charge a margin of 17% on calls that cost 2c/min or less, scaling linearly up to 25% for calls that cost us 15c/min or more, and 25% from there on up.
However, we find that simple phone agents do well on a stack that costs about 2c/min, and that more complex cases typically land around 5-10 c/min - significantly cheaper than most alternatives.
Zeus is brand-new, so we’re eager to get feedback on how it works for customers, and any areas we could improve it. Give it a try and see what you think - let me know if you have any feedback or questions!
No comments yet.