GitHub - 927tanmay/react-ai-voice-avatar: Open-source alternative to real-time avatar APIs like HeyGen Interactive Avatar and Tavus. A talking, lip-synced 3D avatar that runs on your user's GPU, driven by your own LLM. No video stream, no per-minute billing. Or voice mode alone, ChatGPT-style, as a headless React hook. 📦 npm: react-ai-voice-avatar

The open-source alternative to real-time avatar APIs, as a React component.

HeyGen Interactive Avatar, Tavus and Soul Machines sell a talking avatar that holds a conversation, rendered on their servers and billed per streaming minute. This does the same job as an npm install: the avatar renders on your user's GPU, so there is no video stream, no per-minute cost, and no third party in the middle of your conversations.

You bring the model. Point it at OpenAI, Anthropic, your own fine-tune or your existing chat endpoint, and keep your keys on your own backend. The package owns the parts that are tedious to build and easy to get wrong: microphone capture, knowing when someone has finished speaking, interrupting the avatar mid-sentence when they talk over it, streaming speech synthesis, and driving 52 ARKit facial blendshapes at 60 FPS so the mouth matches the words.

It can also run with no backend at all. Speech recognition, generation and voice all have in-browser implementations, which makes for a convincing demo and a genuinely offline kiosk. Most production apps will use their own model and keep only speech and lip-sync on the device.

Don't need a face? The same engine is a headless React hook for voice mode, the way ChatGPT and Gemini do it: react-ai-voice-avatar/headless, with no three.js in your bundle. See it ➔ · How ➔

🌐 Try the live demo ➔ · Voice only ➔

Speech recognition, voice synthesis and lip-sync, all running in your browser tab.

The first visit downloads about 590 MB of models, roughly a minute and a half on a 50 Mbit/s connection. Your browser keeps them, so a return visit is talking in about 2 seconds with nothing downloaded. Replies on the demo come from a hosted model on Groq's free tier, shared by everyone trying it, and start about 3 seconds after you stop talking; on a busy day that tier can run out, and examples/groq-voice runs the same thing on your own free key.

🧪 Status: beta

The engine works and is tested on every push, but it has been checked on far fewer browsers and devices than it is meant for. Here is exactly where things stand, so you can decide what to verify yourself before shipping.

Tested

• Desktop Chrome on macOS (Apple M4, WebGPU), by hand: the live demo (local speech, hosted replies) and the fully local setup.
• Headless Chromium on Linux, in CI on every push: whole turns through your own speech-to-text, model and voice, with stand-ins for the providers; real speech through a fake microphone for push-to-talk; empty replies; sentences kept in order; the in-browser model speaking in your voice; and that no model downloads when you supply your own.

Not tested yet

• Safari on Mac, iPhone and iPad, and Firefox.
• Real Android phones and iPhones: speed, memory, and the microphone.
• Cloud speech against real providers, beyond the stand-ins in CI.
• Long sessions. Nothing has yet held a conversation for 30 minutes and checked memory and turn-taking along the way.

The iPhone behaviour (the smaller MMS voice, and the guard that stops a tab reloading into the same crash) is built around Safari's known memory limits and has not been run on a device. The compatibility and hardware tables mark what is tested and what is expected.

Versions. It is 0.x: a minor version can change behaviour. Since 0.6.0 each CHANGELOG entry says at the top whether it breaks anything. Pin an exact version in production. If it fails on your device, an issue with the browser, the device and the onError output is the most useful thing you can send.

🧩 Built with it

Interview Room, a spoken mock interviewer I built on this package for the DEV Hacktoberfest challenge. Everything runs in the browser: Whisper, Gemma, Kokoro and the avatar. It uses the headless hook for its phone screen, the avatar for its video interview, onSubmit with its own Gemma worker, and speechMs for speaking pace. Live demo ➔

🌟 Two Entry Points (How to use it)

react-ai-voice-avatar provides two distinct ways to integrate into your app depending on your design needs. Both share the exact same underlying conversational state machine, Voice Activity Detection (VAD), and turn-taking logic.

🎧 Entry 1: The Headless Hook ("Voice Mode for your App")

Voice mode for your app: talk to it the way you talk to ChatGPT or Gemini, and interrupt it mid-sentence. The hook owns the microphone, knowing when someone has finished speaking, interruption, and streaming the reply into speech. You own the UI, and every frame it hands you a loudness level to animate.

Try it ➔: a page built on this hook alone. examples/voice-only is the same page as an app to copy.

Import it from react-ai-voice-avatar/headless and Three.js never enters your module graph. Measured on the same Next.js App Router build, one route rendering the 3D avatar and one rendering only the hook:

npm install react-ai-voice-avatar

import { useRef, useState } from 'react';
// The /headless subpath is what keeps Three.js out of your bundle.
// Importing the hook from the package root pulls the 3D stack in with it.
import { useAiVoiceAvatar } from 'react-ai-voice-avatar/headless';

export function VoiceMode() {
  const orb = useRef<HTMLDivElement>(null);
  const [micOn, setMicOn] = useState(false);

  const voice = useAiVoiceAvatar({
    // Your model. Return a string, or a stream so speech starts sooner.
    onSubmit: text => fetch('/api/chat', { method: 'POST', body: text }).then(r => r.body),

    // 0 to 1 every frame, from whoever is talking. Written straight to the
    // DOM: routing it through state would re-render sixty times a second.
    onAudioLevelChange: level => {
      if (orb.current) orb.current.style.transform = `scale(${1 + level * 0.3})`;
    },
  });

  // The mic stays open across turns, so it is tracked apart from `status`,
  // which moves through listening, thinking and speaking.
  const toggle = () => {
    if (micOn) {
      voice.stopListening();
      voice.interrupt();
    } else {
      voice.startListening(); // From a click: this is where the mic prompt appears.
    }
    setMicOn(!micOn);
  };

  return (
    <>
      <div ref={orb} className="orb" data-status={voice.status} />
      <button onClick={toggle} disabled={!voice.isReady}>
        {voice.isReady ? (micOn ? 'Stop' : 'Talk') : 'Loading…'}
      </button>
    </>
  );
}

For captions, onTranscriptUpdate(text, 'user') gives what the user said and onSpeechStart(text) each sentence of the reply as it starts playing. speak(text) says something without a model turn, such as a greeting, and sendText(text) takes typed input. Everything it returns is in the hook reference.

🚀 Voice mode in production

Each stage runs in the browser or on your backend, and the choice is mostly about the download:

Cloud speech is the setup to reach for on phones and on pages people visit once: it is ready as soon as the page is. Local speech keeps what people say on their device and costs nothing per minute, and suits an app they come back to, since the models are stored after the first visit. Fully local is for kiosks and offline use. Use loadModels to hold any download until someone actually engages.

☁️ Cloud Adapters

onTranscribe and onSynthesize replace the local speech models with your own providers. Each replaces its model entirely: supply one and that model is never downloaded. Drop it later and the local model loads, so a provider outage can fall back to the browser mid-session.

To have that fallback ready rather than downloading it at the moment you need it, pass preloadLocalSpeech: the local hearing and voice download behind the conversation, and onLocalSpeechReady fires when they could take over. examples/groq-voice starts on Groq and offers the switch then.

const voice = useAiVoiceAvatar({
  // Instead of Whisper: 16 kHz mono samples of one utterance, to any
  // speech-to-text service. Encode them as WAV if the service wants a file.
  onTranscribe: async (samples: Float32Array) => {
    return await mySpeechToText(samples);
  },

  // Instead of Kokoro: return an encoded MP3/WAV (ArrayBuffer), which the
  // hook decodes, or raw 24 kHz PCM as a Float32Array. Called a sentence at
  // a time, so the first plays while the rest are fetched.
  onSynthesize: async (text: string) => {
    const res = await fetch('/api/tts', { method: 'POST', body: text });
    return await res.arrayBuffer();
  },

  onSubmit: async (text) => fetch('/api/chat', { method: 'POST', body: text }).then(r => r.body),
});

onSynthesize can return audio in whichever form your voice service gives it:

Most raw-PCM voice APIs return 16-bit samples at 16 or 22.05 kHz, so return them with their rate. Bare samples at the wrong rate play too fast or too slow:

onSynthesize: async (text) => {
  const res = await fetch('/api/tts', { method: 'POST', body: text });
  return { audio: new Int16Array(await res.arrayBuffer()), sampleRate: 22050 };
},

The same adapters work on the 3D <AiVoiceAvatar> as props.

🧑‍💼 Entry 2: The Full 3D Avatar

If you want the full visual presence with 60FPS ARKit lip-syncing, use the drop-in 3D component. Under the hood, this is just a wrapper around the useAiVoiceAvatar hook that procedurally maps the audio to a 3D model!

📦 View Package on the Official NPM Registry ➔

# If using the 3D Avatar, you must also install the Three.js ecosystem
npm install react-ai-voice-avatar three @react-three/fiber @react-three/drei

npm install @react-three/drei@^9 @react-three/fiber@^8

npm install react-ai-voice-avatar three @react-three/fiber @react-three/drei --legacy-peer-deps

npm install react@~19.2.0 react-dom@~19.2.0

⚡ DX & Performance (Lazy Code-Splitting)

To prevent the massive ML assets (WebGPU workers, 3D engines) from bloating your initial page load, use the built-in lazy wrapper. It will automatically code-split the 3D dependencies and render a sleek holographic Skeleton UI while the assets download in the background!

import { AiVoiceAvatarLazy } from 'react-ai-voice-avatar';

// Use it exactly like the normal component!
<AiVoiceAvatarLazy avatarPreset="ananya" />

Show the avatar before anyone commits to a download

Lazy loading defers the 3D engine. The speech and language models are the larger cost, and by default they start downloading as soon as the avatar mounts. On a landing page most visitors only look, so render the avatar idle and load the models when someone actually engages:

const [engaged, setEngaged] = useState(false);

<AiVoiceAvatar loadModels={engaged} hideStatusPill={!engaged} />
<button onClick={() => setEngaged(true)}>Talk to it</button>

The live demo's landing page works this way.

📐 Architectural Best Practices

• Standard Import (AiVoiceAvatar): Recommended for full-screen applications where the avatar is the primary product (e.g., Kiosks, Digital Tutors). The browser starts downloading the 3D canvas and ML models straight away, so the avatar is ready as early as it can be.
• Lazy Import (AiVoiceAvatarLazy): Recommended for widgets, modals, or sub-routes (e.g., a "Support Desk" chat bubble in a SaaS dashboard). Defers downloading the 1.5MB 3D engine and WebWorkers until the user actually opens the widget.

⚙️ Server Configuration (Optional Performance Boost)

The react-ai-voice-avatar engine is truly zero-config. You do not need to configure Vite optimizeDeps, Next.js Webpack overrides, or manually host Web Worker files—everything is dynamically bundled and executed automatically!

However, because our ONNX WebGPU engine leverages modern multi-threaded SharedArrayBuffer memory pipelines for maximum inference speed, your hosting server can optionally emit standard Cross-Origin Isolation HTTP headers (COOP/COEP) to unlock peak performance. If these headers are not present, the engine automatically falls back to single-threaded WebAssembly without crashing.

🌐 Enabling Multi-threading on Production (Vercel, Netlify & Cloudflare)

To unlock multi-threaded performance, specify these isolation headers in your routing manifests:

• Vercel (vercel.json): Add "headers": [{ "source": "/(.*)", "headers": [{ "key": "Cross-Origin-Opener-Policy", "value": "same-origin" }, { "key": "Cross-Origin-Embedder-Policy", "value": "require-corp" }] }].
• Netlify / Cloudflare Pages (_headers or netlify.toml): Add /*\n Cross-Origin-Opener-Policy: same-origin\n Cross-Origin-Embedder-Policy: require-corp to public/_headers.

⚡ Enabling Multi-threading in Local Dev (Vite & Next.js)

Vite (vite.config.ts):

export default defineConfig({
  plugins: [react()],
  server: {
    headers: {
      'Cross-Origin-Opener-Policy': 'same-origin',
      'Cross-Origin-Embedder-Policy': 'require-corp',
    },
  },
});

Next.js (next.config.mjs):

export default {
  async headers() {
    return [
      {
        source: '/(.*)',
        headers: [
          { key: 'Cross-Origin-Opener-Policy', value: 'same-origin' },
          { key: 'Cross-Origin-Embedder-Policy', value: 'require-corp' },
        ],
      },
    ];
  },
};

Webpack 5 and Next.js: the "Critical dependency" warnings

Built with webpack, an app using the package shows four warnings like this one, and webpack-dev-server puts them in an overlay over the page:

WARNING in ./node_modules/onnxruntime-web/dist/ort.min.js
Critical dependency: require function is used in a way in which dependencies cannot be statically extracted

They come from a branch of onnxruntime-web that only runs in Node, and are harmless. To hide them, in webpack.config.js:

module.exports = {
  // ...
  ignoreWarnings: [{ module: /onnxruntime-web/, message: /Critical dependency/ }],
};

Or in next.config.ts, for a Next.js build that uses webpack (next build --webpack, or Next.js 15 and earlier; Turbopack does not show them):

const nextConfig: NextConfig = {
  webpack: config => {
    config.ignoreWarnings = [
      ...(config.ignoreWarnings ?? []),
      { module: /onnxruntime-web/, message: /Critical dependency/ },
    ];
    return config;
  },
};

⚡ 3D Avatar Quickstart

import React, { useRef, useState } from 'react';
import { Canvas } from '@react-three/fiber';
import { OrbitControls } from '@react-three/drei';
import { AiVoiceAvatar, type AiVoiceAvatarHandle } from 'react-ai-voice-avatar';

export function App() {
  const avatarRef = useRef<AiVoiceAvatarHandle>(null);
  const [text, setText] = useState('');

  return (
    <div style={{ width: '100vw', height: '100vh', position: 'relative' }}>
      <Canvas camera={{ position: [0, 0.15, 2.2], fov: 32 }}>
        <color attach="background" args={['#101116']} />
        
        {/* Subtle studio lighting */}
        <pointLight position={[-3, 2, -2]} intensity={25} color="#E67E22" distance={6} />
        <pointLight position={[3, 1, -2]} intensity={20} color="#2980B9" distance={6} />
        
        <OrbitControls target={[0, 0.05, 0]} />
        
        {/* Connected Brain: your model answers, so no language model is downloaded */}
        <AiVoiceAvatar
          ref={avatarRef}
          avatarPreset="ananya"
          lightingPreset="studio"
          ttsEngine="kokoro"
          ttsVoice="af_heart"
          // Connect your backend here (receives user speech transcript):
          onSubmit={async (text) => {
            const res = await fetch('/api/chat', { 
              method: 'POST', 
              body: JSON.stringify({ prompt: text }) 
            });
            return res.body; // Avatar natively reads streams!
          }}
        />
      </Canvas>

      {/* Fallback Text Input for noisy environments */}
      <form
        onSubmit={(e) => {
          e.preventDefault();
          if (text.trim() && avatarRef.current) {
            avatarRef.current.sendText(text);
            setText('');
          }
        }}
        style={{ position: 'absolute', bottom: '20px', left: '50%', transform: 'translateX(-50%)', display: 'flex', gap: '8px', zIndex: 100 }}
      >
        <input 
          value={text} 
          onChange={e => setText(e.target.value)} 
          placeholder="Type a message..." 
          style={{ padding: '8px 16px', borderRadius: '20px', border: 'none', background: 'rgba(255,255,255,0.9)', width: '300px' }}
        />
        <button type="submit" style={{ padding: '8px 16px', borderRadius: '20px', border: 'none', background: '#3b82f6', color: 'white', cursor: 'pointer' }}>
          Send
        </button>
      </form>
    </div>
  );
}

🔌 Connect Your Backend (Recipes)

The onSubmit prop natively accepts a string, an AsyncIterable<string>, or a ReadableStream. To connect your actual backend, simply drop in one of these copy-paste recipes to parse your streaming format!

Recipe 0: Plain Text Stream (Fastest & Simplest)

If your backend uses Vercel AI SDK's streamText(...).toTextStreamResponse() or otherwise streams plain raw text, you can pass the stream natively without any parsing!

onSubmit={async (text) => {
  const res = await fetch('/api/chat', { method: 'POST', body: JSON.stringify({ prompt: text }) });
  return res.body; // Natively supported! Returning `res` itself works too.
}}

Text is cut into sentences as it arrives, so the voice starts on the first one. Decimals, abbreviations and initials stay whole, and Hindi (।), Urdu (۔ ؟) and Chinese or Japanese (。！？) sentence endings are recognised. A sentence shorter than 35 characters waits to join the next one, unless the stream goes quiet for half a second, so a "Sure." before a slow tool call is still said straight away.

Recipe 1: Vercel AI SDK (≤v4 Data Stream)

Older versions of the Vercel AI SDK stream data using a specific protocol (e.g., 0:"Hello"). This recipe parses those chunks into clean text with a carry-over buffer for safe network boundaries.

onSubmit={async function* (text) {
  const res = await fetch('/api/chat', { method: 'POST', body: JSON.stringify({ prompt: text }) });
  if (!res.body) return;
  const reader = res.body.getReader();
  const decoder = new TextDecoder();
  let buffer = '';
  
  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    buffer += decoder.decode(value, { stream: true });
    const lines = buffer.split('\n');
    buffer = lines.pop() ?? ''; // keep the trailing partial chunk
    for (const line of lines) {
      if (line.startsWith('0:')) {
        try { yield JSON.parse(line.substring(2)); } catch { /* ignore keep-alive / non-JSON frames */ }
      }
    }
  }
}}

Recipe 2: OpenAI-Compatible SSE Endpoint (and AI SDK v5)

Standard Server-Sent Events (SSE) stream data: {...} blocks. This handles safe parsing across broken network chunk boundaries.

onSubmit={async function* (text) {
  const res = await fetch('/api/chat', { method: 'POST', body: JSON.stringify({ prompt: text }) });
  if (!res.body) return;
  const reader = res.body.getReader();
  const decoder = new TextDecoder();
  let buffer = '';
  
  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    buffer += decoder.decode(value, { stream: true });
    const lines = buffer.split('\n');
    buffer = lines.pop() ?? ''; // keep the trailing partial chunk
    for (const line of lines) {
      if (line.startsWith('data: ') && line !== 'data: [DONE]') {
        try {
          const parsed = JSON.parse(line.substring(6));
          // AI SDK v5 emits {type:'text-delta', delta:'...'}; OpenAI emits choices[0].delta.content
          if (parsed.type === 'text-delta' && parsed.delta) {
            yield parsed.delta;
          } else if (parsed.choices?.[0]?.delta?.content) {
            yield parsed.choices[0].delta.content;
          }
        } catch { /* ignore keep-alive / non-JSON frames */ }
      }
    }
  }
}}

Staying quiet, and how long they spoke

Return an empty string (or a stream that ends without text) and the avatar says nothing: the turn ends and it keeps listening. That lets you gather one answer across several pauses, replying only once the user has finished.

The second argument says how long the user spoke, from their first word to their last. The padding kept from before speech and the silence waited out after it are left out, so it is the figure for a speaking rate. It is undefined for text sent with sendText.

onSubmit={async (text, { speechMs }) => {
  answer += ' ' + text;
  if (!answerIsComplete(answer)) return ''; // say nothing, keep listening
  const wpm = speechMs ? text.split(/\s+/).length / (speechMs / 60000) : undefined;
  return askMyBackend(answer, { wpm });
}}

Stopping a reply

A reply belongs to one turn. When the user talks over it, interrupt() is called, or a new question arrives, the hook stops reading it and aborts signal. Pass the signal to fetch so the request, and any model run or tool call behind it, stops too:

onSubmit={async (text, { signal }) => {
  const res = await fetch('/api/chat', { method: 'POST', body: JSON.stringify({ text }), signal });
  return res.body;
}}

A streamed reply may go quiet for as long as it needs, for a tool call for instance. The hook waits while the stream is open; it is your stream's job to end, or to fail, if the work behind it hangs.

🗣️ Languages

English and Hindi both run entirely on the device. Set ttsLanguage and the engine picks a matching voice, a matching speech-recognition model, and a matching phoneme path.

<AiVoiceAvatar ttsLanguage="hi-IN" />

Hindi needed real work rather than a config flag. Kokoro ships four Hindi voices inside the same checkpoint as the English ones, but the JavaScript wrapper does not list them, and the bundled eSpeak build carries English data only and rejects hi outright. So this package includes its own Devanagari-to-phoneme converter. Devanagari is close to phonemic, which makes that tractable; the hard part is schwa deletion, the rule that makes कमल read as "kamal" rather than "kamala", and getting it wrong produces speech that still sounds like speech while sounding like someone spelling Hindi out.

Because the converter emits real phonemes, Hindi gets phoneme-driven lip-sync with distinct mouth shapes for the retroflex consonants, not the amplitude-only mouth flapping most engines fall back to outside English. Code-switching works too: English words inside a Hindi sentence are routed to the English phonemiser, so "मुझे coffee चाहिए" is pronounced correctly throughout.

A session is one language at a time. Someone who switches language mid-conversation will be mistranscribed, because the recognition model is told which language to expect and browser Whisper cannot detect it.

🎤 Taking turns

The user can talk over the avatar and cut it off mid-sentence. This is on by default, because waiting for a reply to finish is the thing that makes a voice agent feel like a walkie-talkie.

<AiVoiceAvatar
  allowInterruption={false}   // default: true
  onUserInterrupt={() => analytics.track('barge_in')}
/>

Turn it off for a kiosk or a noisy room, where the avatar hearing its own voice through the speakers and stopping itself is worse than waiting. It is ignored in push-to-talk, which owns the floor explicitly.

Push-to-talk

With listenMode="push-to-talk" the button decides when a turn ends, not a silence. Call startListening() when it goes down and stopListening() when it comes up: what was said in between is handed over as one turn, however long the pauses in it. Pressing during a reply stops the reply. interrupt() abandons a hold without sending it, and a hold with no speech in it sends nothing.

<button
  onPointerDown={() => voice.startListening()}
  onPointerUp={() => voice.stopListening()}
>
  Hold to talk
</button>

Stopping and muting

Three controls, for three different intentions:

So a Stop button that should silence everything calls both:

voice.interrupt();
voice.stopListening();

A mute button, like the one in a call, calls setMicMuted(!voice.isMicMuted). Unmuting needs no new permission prompt.

How a turn is decided

Voice detection scores every 96ms frame for how much it sounds like speech. A cough, a door and a chair all clear a loudness bar as easily as a word does, so loudness alone cannot separate them — sustain can. A sound has to keep scoring as speech for minSpeechMs before the engine treats it as a turn.

Below that bar, a sound only ducks the avatar's voice, reversibly. If it turns out to be a cough the reply resumes from where it paused, at the right place in the sentence and with the mouth still in sync. Nothing about the conversation changed, because nothing was decided on a noise.

Tuning for your room

The defaults suit a quiet room and headphones. A shop floor is a different problem, and only you know which you have.

<AiVoiceAvatar
  speechDetection={{ positiveSpeechThreshold: 0.6, minSpeechMs: 700 }}
/>

Two costs worth knowing before you change anything. A speaker who sits below positiveSpeechThreshold produces no events at all — raising it trades quiet voices for quiet rooms. And a filler like "hmm" held long enough to pass minSpeechMs still reaches transcription; a denylist catches the common ones after the fact, but sustain cannot tell a long "hmm" from a short word.

🚨 Handling failures

Pass onError and your app learns when something breaks, rather than finding out from a console message it cannot see.

<AiVoiceAvatar
  onError={(e) => {
    if (e.severity === 'fatal') showFallbackUI(e.stage);
    logToSentry(e);
  }}
/>

Check severity before reacting. Most failures here are survivable because the engine falls back: WebGPU to WASM, Kokoro to a smaller voice model. Those arrive as degraded and the avatar still works, so treating them as fatal would hide a working experience behind an error screen. A refused microphone is fatal for listening while typed input still works, which is a judgement only your app can make.

stage is one of microphone, speech-recognition, language-model, speech-synthesis, audio-output, worker, model-storage or conversation. There is also a detail string carrying the engine's internal stage name for bug reports; it is not stable across versions, so do not branch on it.

model-storage is always degraded: a model loaded and works, but could not be kept, so the next visit downloads it again. The message says why — usually not enough free space, which on a phone is the common case.

Your own adapters report here too. When onTranscribe, onSubmit or onSynthesize throws, the turn ends, the avatar goes back to listening, and onError gets a degraded report whose detail is the callback's name. When someone was heard but no words were recognised, detail is no-speech. These four detail values are stable, so you can branch on them:

onError={(e) => {
  if (e.detail === 'no-speech') say("Sorry, I didn't catch that.");
  else if (e.detail === 'onSubmit') toast("Couldn't reach the server. Try again?");
}}

Models are kept between visits

Downloaded models are stored in the browser's Origin Private File System, so a returning visitor skips the download: on an Apple M4 with WebGPU, a return visit is ready to talk in about 2 seconds, with nothing fetched. The browser's Cache API — what the model loader uses by default — refuses any single file of 256 MiB or more, and both the voice (~310 MB) and the local language model (~750 MB) are larger than that, so before this they were downloaded again on every visit. Where OPFS is unavailable the Cache API is used as before.

Browsers can clear this storage when disk space runs low. If your app is one people return to, call navigator.storage.persist() from a user gesture to ask the browser to keep it; the engine does not, because Firefox can answer that call with a permission prompt, and a library should not put one in front of your visitors unasked.

Keeping your own models

A model of your own that is over 256 MiB has the same problem. The store the engine uses is its own entry point, with no React and nothing else in it, so a worker that loads your model can import it:

import { env } from '@huggingface/transformers';
import { createModelCache, isModelCacheSupported } from 'react-ai-voice-avatar/model-cache';

if (isModelCacheSupported()) {
  env.useCustomCache = true;
  env.customCache = createModelCache();
}

It is an ordinary match / put / delete cache, so anything that loads through one can use it. onStoreFailed(url, reason) reports a file that could not be kept, such as when the disk is full; the model still loads, and is downloaded again next visit.

🎨 Bring Your Own 3D Avatar (Custom GLB)

You are not locked into our built-in avatars (ananya and aarav)! You can use any custom .glb humanoid model by passing its URL or local path to the modelSrc prop:

<AiVoiceAvatar
  modelSrc="/models/my-custom-avatar.glb"
  // ...
/>

📋 Custom Avatar Requirements

To ensure the lip-sync and procedural facial dynamics engines work correctly, your custom model must meet the following standard requirements:

1. Format: .glb (GLTF Binary).
2. Facial Blendshapes (Morph Targets): The model's head/face mesh must contain the standard 52 Apple ARKit blendshapes (e.g., jawOpen, eyeBlinkLeft, mouthSmileRight). Our engine automatically traverses your model to find these targets.
3. Bone Naming: For the interactive mouse-tracking and head-tilting physics to function, the armature should use standard bone names (e.g., a neck/head bone named Head, head, Neck, or neck).

Where the built-in avatars come from

ananya and aarav are converted from the Microsoft Rocketbox library, which is MIT licensed like this project, so you can redistribute them without restriction. The library has 115 avatars and any of them can be converted with scripts/convert-rocketbox.py. See assets/avatars/LICENSE.md.

Models exported from Ready Player Me also work, since they carry the same ARKit blendshapes and bone names. Note that Ready Player Me shut down in January 2026, so you can no longer create new avatars there, and existing exports are licensed CC BY-NC-SA rather than MIT.

🏗️ What runs where

Four stages, and you choose where each one happens. The defaults are all local, which is why the demo needs no keys, but the interesting production setups are mixed.

flowchart TD
  mic(["🎙️ You speak"]) --> vad["Voice detector<br/><i>in the browser</i>"]
  vad --> hear["Hearing<br/><i>Whisper in the browser</i><br/>or your onTranscribe"]
  hear --> think["Thinking<br/><i>a model in the browser</i><br/>or your onSubmit"]
  think --> speak["Speaking<br/><i>Kokoro in the browser</i><br/>or your onSynthesize"]
  speak --> face["🔊 Voice, with a lip-synced<br/>3D face or your own UI"]
  vad -. "talk over it and it stops" .-> speak

Each stage runs in the browser unless you pass the adapter beside it, which hands that stage to your own service.

The face never leaves the device, which is the whole point: that is what avatar APIs charge per minute for, and it is the one stage that cannot be outsourced without a video stream.

The common production shape is onSubmit alone. Hearing and speaking stay local, so no audio ever leaves the browser, while generation goes to whatever model you already run. That keeps the download to about 600 MB (Whisper base and the Kokoro voice), keeps your keys on your server, and still gives you a conversation nobody else can read.

Add onTranscribe and onSynthesize as well and nothing downloads at all, apart from the ~4 MB voice detector once the microphone opens. Each adapter replaces its model entirely, so the page is ready as soon as it loads. That is the shape for phones and for pages people visit once.

Fully local is real, not a demo trick, and it is the right answer for a kiosk, a regulated environment, or anywhere without reliable connectivity. Be aware of the cost: in English the first visit downloads roughly 1.3 GB before anyone can speak (the language model alone is 750 MB), about 2.1 GB in Hindi, and the quality ceiling is whatever a model that size can do.

⏱️ How fast it answers

Measured with scripts/measure-latency.mjs on an Apple M4 with WebGPU in Chromium: a spoken question through a fake microphone, the median of five turns after a warm-up turn, timed by the hook's own callbacks.

So from the moment you stop talking to the first word of the reply:

The hosted figure adds the route's round trip to the measured speech stages, since that route answers only the live site. Most of the wait is the pause for silence, which is what stops the avatar cutting people off while they think. Lower redemptionMs in speechDetection for a snappier feel, at the cost of answering half-finished sentences. Without WebGPU, recognition and the voice run on the CPU and are several times slower.

🔀 Hosted now, local when warm

You do not have to choose once. preloadLocalLlm fetches the in-browser model behind the conversation while your backend answers, and onLocalLlmReady tells you when it can take over:

const [localReady, setLocalReady] = useState(false);

<AiVoiceAvatar
  // Dropping onSubmit is what hands the conversation over.
  onSubmit={localReady ? undefined : askMyBackend}
  preloadLocalLlm
  onLocalLlmReady={() => setLocalReady(true)}
/>

Nobody waits for a gigabyte to say the first word, and the turns your backend answered are handed to the local model, so it does not restart the conversation from nothing. Useful for a kiosk that must keep working when the wifi drops, a demo on someone else's quota, or a phone, which will never accept the download but can hold a conversation the moment it loads.

The live demo runs exactly this: replies come from a hosted model, and on a desktop with WebGPU the in-browser model downloads during the conversation and takes the floor when it lands. The page says which one is answering at any moment.

Explore the canonical patterns in the examples/ directory:

📖 API Reference

<AiVoiceAvatar /> Props

Imperative Ref API (AiVoiceAvatarHandle)

Attach a React ref (useRef<AiVoiceAvatarHandle>(null)) to access imperative real-time controls:

interface AiVoiceAvatarHandle {
  /** Command the 3D avatar to speak an arbitrary string with synchronized acoustic lip blending */
  speak: (text: string) => void;
  /** Manually engage microphone recording and Voice Activity Detection (VAD) */
  startListening: () => void;
  /** Pause active microphone listening */
  stopListening: () => void;
  /** Mute or unmute the microphone without closing it; replies keep playing */
  setMicMuted: (muted: boolean) => void;
  isMicMuted: boolean;
  /** Instantly interrupt and halt active voice speech synthesis and clear the audio queue */
  interrupt: () => void;
  /** Manually submit text to the onSubmit handler, simulating a spoken utterance (useful for text-only fallback) */
  sendText: (text: string) => void;
  /** Wipe multi-turn conversation memory history and caption overlay states */
  clearHistory: () => void;
  /** Retrieve live Web Audio API AnalyserNode powering real-time spectral lip sync */
  getAnalyser: () => AnalyserNode | undefined;
}

💬 Text-Only Input (sendText)

If your users cannot use a microphone (e.g., noisy environments, privacy concerns, or lack of permissions), you can easily wire up a standard text input field to bypass the speech recognition pipeline entirely!

Simply attach a ref and call sendText() to pass a string directly to your onSubmit handler (or local LLM):

const avatarRef = useRef<AiVoiceAvatarHandle>(null);

// In your UI, attach this to a standard <form> submission:
const handleTextSubmit = (userInput: string) => {
  avatarRef.current?.sendText(userInput);
}

When you use sendText, the avatar immediately enters the thinking state and processes the interaction exactly as if the user had spoken it aloud.

useAiVoiceAvatar() (headless)

import { useAiVoiceAvatar } from 'react-ai-voice-avatar/headless';

Takes the same options as the component's props, except the ones about the 3D scene and its overlays (avatarPreset, avatarSize, modelSrc, lightingPreset, gestures, showCaptions, hideStatusPill, onModelLoaded, debug and the styling props). onStatusChange is replaced by the returned status. A few options matter mostly without an avatar:

It returns:

It also returns several refs (currentSpeechTextRef, audioContextRef and others) that the 3D component uses for lip-sync. A voice UI can ignore them.

🌐 Performance & Asset Caching

1. Native WebGPU & WASM Degradation:
   • Modern Chromium browsers (Chrome, Edge, Opera, Arc) on desktop and mobile platforms benefit from hardware-accelerated WebGPU neural execution.
   • On systems without WebGPU, inference falls back to WebAssembly (WASM), which is slower.
2. Persistent Local Caching:
   • AI models (Whisper, Kokoro, the local language model) are downloaded once and kept in the browser's origin private file system, so later visits skip the download. See Models are kept between visits.

🤝 Contributing & Open Issues Roadmap

We actively welcome community contributions. CONTRIBUTING.md has the local development guide, and ROADMAP.md has what is worth doing next, why, and what is already known about each item — including the measurements behind the open questions.

The largest pieces currently open:

1. 🖐️ Open-palm gestures. Hands now gesture while the avatar speaks, but always with the palms facing inward. Turning them up — the open, offering gesture people use when explaining — needs the forearm's roll, which has to be worked out from the finger bones rather than assumed, for the same reason as everything else in armRig.ts.
2. 🎤 Turn-taking in real rooms. Turns are tuned against one speaker in a quiet room. A short "yes" can be dropped, and a quiet speaker can go unheard. Recordings from more voices and noisier rooms would let the defaults be set against something other than one person.
3. 🎭 Expanding regional 3D avatar personas. Ananya and Aarav ship out of the box. Royalty-free character GLBs (~3MB) rigged with the standard 52 Apple ARKit facial blendshapes are welcome. Avatar meshes are served from a CDN rather than bundled, so adding one adds nothing to the npm install (2.3 MB tarball, 5.8 MB unpacked, asserted in CI by scripts/verify-pack.mjs).
4. 🙌 Gestures. The avatar stands still while it speaks. Hand and arm movement tied to speech is the most visible thing still missing.
5. 📱 React Native / Expo support. Exploring bindings to run ONNX inference and Three.js on mobile runtimes.

Hindi speech and VAD sensitivity tuning (speechDetection) were previously listed here and have both shipped.

🧭 Browser Compatibility Matrix

The library relies on WebGL, Web Audio, Web Workers and, where available, WebGPU. Where WebGPU is missing the models run on WebAssembly instead, which is slower. Only desktop Chrome has been tested. The rest of this table is what the code is built to do, not what has been seen on that browser.

Tried it on one of these? An issue saying what worked and what didn't fills in this table.

💻 Hardware Requirements

Running neural networks in the browser needs capable hardware. The figures below are estimates: only an Apple M4 MacBook has been measured, and no phone has been tried yet.

📜 License

MIT © React AI Voice Avatar Contributors.