Google Chrome downloads a new neural AEC while you are not looking

If you use Google Chrome on desktop, a 425 KB TFLite model has landed in your browser profile and has taken over part of the acoustic echo cancellation. It runs as part of the pipeline used by every getUserMedia({audio: true}) call, default-on with no constraint and no flag. Available to all web pages in Google Chrome, but not in other browsers. Not even Chromium-based ones.

The model shows a median 5.4 dB of extra echo suppression on a hard echo corpus, and it halves the word error rate a transcribing voice agent gets on the near-end talker. Sounds amazing, right?

While preparing WebRTC Insights #150 I had to write something about a Chromium issue regarding getUserMedia initialization delay:

ML-based residual echo estimation was launched in b/442444736. The feature introduced an avoidable delay on the getUserMedia, from allocating TF Lite structures and tensors.

The latency is a small regression of the feature it cites, which had already launched. I had been watching that issue, but not closely. Digging in was quite interesting.

How echo happens, and how AEC3 handles it

WebRTC’s AEC3 echo cancellation got a post here back when it launched in 2018 – silently. Since we now have LLMs it is much easier to understand and explain how it works.

The far end talks. The audio gets delivered over WebRTC, decoded, and played out through your speaker. It bounces around your room and arrives at your microphone mixed with your own voice. You send that mixture back and they hear themselves, delayed by the round trip.

We know exactly what we sent to the speaker, so in principle we subtract it. What comes back is that reference through a room impulse response that differs in every room, changes when you tilt your screen, and arrives after an unknown, drifting amount of buffering. That can be modelled with a linear filter. Distortion from an overdriven loudspeaker cannot, since it is nonlinear.

So AEC3 splits the job:

• An adaptive linear filter. Estimate the delay, align the reference, subtract a continuously reshaped copy of it in 4 ms blocks.
• The suppressor, driven by the residual echo estimator, which calculates R2, i.e. how much echo power is still in this bin. The suppressor turns that into a gain between 0 and 1, per bin, per block.

Finding an estimate for R2 is genuinely hard. Pick it too high and you suppress the near-end talker, the half-duplex feel where you cannot interrupt without being chopped up. Pick too low and the far end hears themselves.

The classic estimate is built on one metric: echo return loss enhancement, ERLE, the ratio of microphone power to residual power per band, with patches for the reverb tail, for clipping and for a filter that has not yet converged. We like metrics because we optimize them, and this is the one echo cancellation has been optimized against for a decade. It is also the one number a web application can see, standardized on RTCAudioSourceStats. And it is one scalar per band describing the linear filter, used to predict the nonlinear leftovers.

That estimate is intentionally biased high. Audible echo is a failure every user notices and blames on the product; a chopped-up talker gets blamed on the network. So it is a conservative estimate, and an inflated R2 closes the gain on bins holding your voice rather than echo. Double-talk failure is a consequence of that.

The neural residual echo estimator

The model replaces the residual echo estimator component. It reads the signals directly, carries recurrent state, and emits a per-bin mask, meaning this fraction of the energy here is echo. Multiply the residual spectrum by it and there is R2. A learned model can recognize a distorting speaker or an echo path that just changed.

With a model present, the AEC3 suppressor takes a different configuration, supplied by the estimator itself: reacting per block instead of averaging, thresholds moved toward transparency, allowed to open back up faster. It also stops switching between its refined and coarse filter outputs and always takes the refined one.

Three bands, three layers, 48 units each

Everything in this section is LLM reverse engineering, take it with a grain of salt. I pulled the tflite file out of my Chrome profile and had Claude decode the flatbuffer. None of it is documented, and the internal names inside the file are the only corroboration.

The graph is nine LSTM cells. An LSTM carries a small memory from one frame to the next, a hidden state and a cell state, so what it says about this block depends on what it has been hearing. That is the point: an echo path misbehaves in ways that only become recognizable over time. TFLite offers a ready-made LSTM operator and this file does not use it, spelling every cell out as matrix multiplies, sigmoids and tanhs instead. 321 operators, no convolution, no attention, roughly 256k int8 parameters.

The state shape is layers by branches by (hidden, cell) by units: three layers of three parallel cells of 48. Parallel means a frequency band split. The 129 bins are cut into three bands of 43 and each cell sees only its own.

This deserves applause of a kind machine learning work rarely earns. One 144-unit LSTM needs a 144×144 recurrent matrix; three 48-unit cells need three of 48×48, one third of the cost, and what is given up is cross-band communication that mostly is not needed, because low, mid and high frequencies have their own echo behaviour. That is a subband filter bank, the oldest move in the DSP playbook, applied to the network topology instead of the signal. It is why this fits in 0.16% of a cpu core instead of being a server-side feature.

The recipe is not new to real-time communications. We have seen a linear model combined with an LSTM before, in Meta’s machine-learning bandwidth estimation talk at RTC@Scale 2024. libWebRTC itself still carries traces of the same idea: NetworkStatePredictor, a hook added in 2019 to let a model override one decision in the bandwidth estimator and never implemented (in the opensource version).

Running it on the AEC Challenge corpus

I had Claude run AEC3 twice over the ~500 test files from Microsoft’s AEC challenge synthetic set through libWebRTC’s audioproc_f, once as a baseline and once with Chrome’s model, so every file has two ERLE numbers. In a nutshell this is the script driving the process:

The script writes two wav files per input, which you can listen to, or score using the clean near-end and echo signals the synthetic corpus ships alongside.

The corpus is built to be hard: every file is double-talk, most have a nonlinear far end. Median echo suppression on far-end-only frames goes from 19.8 dB to 27.0, and the median per-file improvement is 5.4 dB, better on 80% of files. Near-end level during double-talk is unchanged, +0.04 dB median. For visualization, files are sorted by the baseline number.

Across the left-hand half, where AEC3 is struggling, the model buys close to 10 dB of additional echo suppression. On the right, where AEC3 was already good, it matches it. How much of it your users get depends on which half of the graph they’re in: laptop speakers on the left, headsets on the right.

Running the AECMOS tool from the same repository shows how a human listener would evaluate it. Better on the left where AEC3 struggled, almost no difference on the right where it was already good.

Both results are consistent over the other 9500 files of the corpus. The full dataset shows that in good conditions (on the far right), the model performs just as well as plain AEC3. In the test set, that right end had looked a bit concerning.

What it does to the words

Tsahi Levent-Levi had a chat with Justin Uberti recently and Justin suggested tuning how the AEC handles double talk which needs to be different for GenAI voice use-cases (around 19:00). Since experiments are cheap now I had Claude try.

Voice agents are a use case nobody was tuning AEC3 for. An agent speaks through the loudspeaker and listens on the microphone at the same time, and what it hears goes to an ASR rather than to a person. So how does the new estimator do there?
Word error rate is not a metric I would have reached for. Claude suggested it (and Krisp used it too recently), and the run took about an hour: 125 files across the ERLE range, Whisper small.en on each, scored against its own transcription of the clean near-end so its quirks cancel out.

On the raw microphone, 16.5% of the words Whisper returns belong to the far end. That number is the argument for never turning AEC off. Plain AEC3 brings it to 0.28% and the model moves it to 0.36%, which is no change.

Baseline ERLE	n	Echo removed	Base WER (median)	Neural WER (median)
0-15 dB	37	+7.56 dB	0.600	0.545
15-22 dB	35	+9.14 dB	0.529	0.235
22-30 dB	36	+1.41 dB	0.426	0.172
30-60 dB	17	+0.07 dB	0.381	0.143

Median word error rate halves with the neural estimator, 0.500 to 0.235, on the same files with the same echo removed by the same linear filter. More of the near-end talker’s speech survives the suppressor intact. A better R2 is doing two jobs:

• Where the linear filter failed, the classic estimate runs low and leaks. A better R2 suppresses seven to ten more decibels of real echo, but those utterances are wrecked anyway.
• Where it worked, there is no echo left to remove, but the estimate still runs high and clamps the gain shut on the talker. A better R2 keeps it open.

Which is what the AECMOS tool shows for the “other degradations”, a less exotic metric than transcription word error rate:

Google Chrome gets it, everyone else is stuck with AEC3 from 2018

The weights arrive at runtime over Chrome’s ML delivery pipeline, the same one used for on-device page classification, and land in your browser profile under optimization_guide_model_store/71/. They are in no repository and carry no license.

Edge is built from the same code, with the flag enabled by default, and gets nothing, because it is not a client of that service. Electron gets nothing for the same reason. Firefox and Safari vendor libwebrtc, so they run AEC3 and do not get it either. The interface for the model is open source, the model itself is not.

AEC3 has been the universal baseline for a decade. Every browser, every Electron app, every native SDK that vendored libwebrtc got the same echo canceller (note: mobile might have preferred the hardware ones on phones). AEC3 will stay with us, but we do not expect new work on it.

Nothing gates by origin at least. Google Meet, Jitsi and a weekend project all get the identical canceller in Google Chrome. But they behave differently when not used in Google Chrome.

It is enabled by default on Chrome desktop, and nothing in getStats() reports which estimator ran. The one stat that potentially moves reports it backwards. echoReturnLossEnhancement is measured on the linear filter, before the suppressor, and with a model present the coarse-filter fallback is skipped, so that residual gets louder. At service level Chrome may look like it cancels half a decibel less than before, or than Edge, while it actually cancels more echo:

Google Meet apparently does not look at those stats.

This gets delivered at runtime, whenever getUserMedia gets called. It is downloaded from an endpoint named https://optimizationguide-pa.googleapis.com/v1:GetModels. If you wonder what the pa stands for the Chromium source code has you covered: “private API”. At least it is 1/1000th the size its Gemini Nano cousin.

Closing thoughts

It was pretty clear for a while that a hand-tuned AEC would be struggling to compete with more modern approaches such as DeepVQE. The results look amazing, as does the architecture and engineering. Oh and Claude did an amazing job at explaining this to me. I am not an audio person but diagrams like the ones in this post (which may or may not exist inside Google) taught me a ton. I haven’t looked back at my initial attempts to use Gemini in a codebase largely written by Googlers for the last six months.

Since libwebrtc was open sourced in 2011, the audio quality floor of the web has been a shared asset. libwebrtc is BSD-licensed, AEC3 has been BSD since it replaced AEC2 around 2018, and everyone shipping it got the same echo canceller: Chrome, Edge, Firefox, every native SDK. Quality differences came from what you built around it.

Why did this ship secretly and Chrome-only? This is not an API and you cannot tell from a single call getStats() whether it is running so one can debate whether this is “web-exposed” and requires the blink launch process. The discussion on the Prompt API suggests that other Chromium downstream vendors were not thrilled with Google’s decisions about shipping models that somehow change behavior. The only paper trail we have is an “analysis and approvals” document linked from the Chromium change which enables neural residual echo estimation in WebRTC by default but it is not public – which is common.

{“author”: “Philipp Hancke“}