Benchmark
Whisper large-v3 is a speech recogniser. As published its weights are 3.09 GB. Our TWX4 build is 0.98 GB, smaller than the published turbo version at 1.62 GB. On nine of ten test sets it adds at most 0.3 points to the original’s error rate. On Danish it adds 1.06.
See the matching eval card for the result in one screen.
The setup
Four builds, ten test sets, one 20 GB card.
Large-v3 is the full model. Large-v3-turbo is its published faster version. We made a TWX4 build of each and measured all four the same way.
Whisper large-v3 as publishedThe reference for every error rate below
3.09 GB
Whisper large-v3-turbo as publishedThe published faster version
1.62 GB
Tileward TWX4 of large-v368% smaller than large-v3 (3.2×)
0.98 GB
Tileward TWX4 of turbo65% smaller than turbo (2.9×)
0.56 GB
Error rate
At most 0.3 points added on nine of ten sets, and 1.06 on Danish.
Error rate is the share of words transcribed wrong, or of characters for Mandarin and Japanese. Lower is better. Each build’s column is its change from the original, in points, with its 95% range.
TWX4 of large-v3: +0.01Turbo as published: +0.11
LibriSpeech test-otherOriginal 3.57%
TWX4 of large-v3: +0.16Turbo as published: +0.36
Synthetic call-centre clipsOriginal 0.79%
TWX4 of large-v3: −0.02Turbo as published: −0.10
FLEURS GermanOriginal 5.93%
TWX4 of large-v3: +0.11Turbo as published: +0.43
FLEURS SpanishOriginal 4.21%
TWX4 of large-v3: +0.09Turbo as published: +0.39
FLEURS HindiOriginal 18.48%
TWX4 of large-v3: +0.16Turbo as published: +2.04
FLEURS ArabicOriginal 16.69%
TWX4 of large-v3: +0.27Turbo as published: +0.57
FLEURS Mandarin, charactersOriginal 6.23%
TWX4 of large-v3: +0.27Turbo as published: +0.49
FLEURS Japanese, charactersOriginal 6.06%
TWX4 of large-v3: −1.22Turbo as published: +0.02
FLEURS DanishOriginal 14.02%
TWX4 of large-v3: +1.06Turbo as published: +2.29
Japanese turns on single clips: the original repeats itself to the length limit on one of them, so read that set as level.
Against turbo
Smaller than turbo, and level with it or ahead on every set.
The large-v3 build is 40% smaller than turbo (1.7×). Compared clip for clip, its error rate is lower on four sets and level on the other six. Turbo is the faster of the two, by about two to one on the same card.
LibriSpeech test-other
TWX4 of large-v3, against turbo: −0.19
FLEURS German
TWX4 of large-v3, against turbo: −0.32
FLEURS Spanish
TWX4 of large-v3, against turbo: −0.31
FLEURS Hindi
TWX4 of large-v3, against turbo: −1.88
The other six sets
Level: each range spans zero
Calls on one card
8% more live calls for large-v3, and 10% more for turbo.
A live call here is one second of audio transcribed per second, on one 20 GB card. We sent 512 windows of 28 seconds of speech at once.
Whisper large-v3 as published3.09 GB weights
140
Tileward TWX4 of large-v30.98 GB weights
1528% more
Whisper large-v3-turbo as published1.62 GB weights
301
Tileward TWX4 of turbo0.56 GB weights
33110% more
Large-v3 fills this card’s memory, so a larger card holds more calls; turbo reaches the server’s limit of 256 requests first.
Turbo in TWX4
65% smaller than turbo, and behind it on four sets.
Our turbo build is 0.56 GB against 1.62 GB. Compared clip for clip with turbo, it is level on six sets and behind on these four, by 0.09 to 0.68 points.
LibriSpeech test-other
TWX4 of turbo, against turbo: +0.09
FLEURS Arabic
TWX4 of turbo, against turbo: +0.68
FLEURS Mandarin, characters
TWX4 of turbo, against turbo: +0.53
FLEURS Japanese, characters
TWX4 of turbo, against turbo: +0.31
Which build
Large-v3 in TWX4 for accuracy, turbo in TWX4 for calls per card.
The large-v3 build adds at most 0.3 points on nine sets, at about a third of the original’s size. The turbo build carries 331 live calls on one 20 GB card, and trails turbo by up to 0.68 points on four sets.
How it was measured
Every build on the same card, the same server and the same clips.
Each error rate is compared with Whisper large-v3 as published, clip for clip.
The card. One 20 GB card. Every build ran on vLLM from its own weights, with the server allowed 90% of the card’s memory.
Size. The weights file. Loaded on the card, the weights take the same space.
Test sets. LibriSpeech test-clean (2,611 clips) and test-other (2,932). FLEURS test in German (841), Spanish (907), Hindi (417), Arabic (428), Mandarin (944), Japanese (650) and Danish (929). And 384 synthetic call-centre clips: scripted call lines in eight synthetic voices, each clean, through a telephone band and with noise.
Scoring. Greedy decoding, with the language given and one 30-second window a clip. Word error rate after Whisper’s own text normaliser, or character error rate for Mandarin and Japanese. Each change has a paired bootstrap 95% range from 2,000 resamples of the clips.
Live calls. 512 windows of continuous read English, 28 seconds each, sent at once, with up to 256 in progress. The figure is seconds of audio transcribed per second of wall time. Each run took 35 to 83 seconds, so start-up and tail are inside it.