Finding 02
Finding 2: Comparing Whisper model sizes
In one sentence: The medium model had the lowest word error rate in this sample; we also measured processing time to compare its hardware cost with the other models.
Choosing a speech model
The choice of speech model affects transcription errors and processing time. We tested six Whisper models on the same recordings to compare those costs.
The phone records and uploads audio. Speech recognition runs on the server. The suggested hardware tiers below refer to servers; all measured timings come from one laptop GPU.
Word error rate (WER) counts substitutions, deletions, and insertions relative to the reference word count. Lower is better. A WER of 0.20 means roughly 1 error for every 5 reference words.
The comparison
| Model | Suggested server tier | Word error rate (WER) | English | Filipino | Transcription time (seconds) |
|---|---|---|---|---|---|
tiny | Minimal CPU box | 0.733 | 0.5862 | 0.8917 | 2.4335 |
base | Small CPU box | 0.559 | 0.3783 | 0.7553 | 2.7327 |
small | Modest server / shared CPU | 0.402 | 0.2799 | 0.5336 | 3.0715 |
medium | Entry GPU server | 0.275 | 0.2284 | 0.3255 | 6.2466 |
large-v3-turbo | GPU server | 0.421 | 0.3884 | 0.4561 | 6.3971 |
large-v3 | GPU server | 0.308 | 0.2497 | 0.3716 | 59.8073 |
The English / Filipino gap
Every model had a higher WER in Filipino than in English. The gap varied by model, as shown below.
| Model | Filipino WER minus English WER |
|---|---|
tiny | +0.305 ⚠️ |
base | +0.377 ⚠️ |
small | +0.254 ⚠️ |
medium | +0.097 |
large-v3-turbo | +0.068 |
large-v3 | +0.122 ⚠️ |
Largest gap: base (+0.377). Smallest gap: large-v3-turbo (+0.068).
The overall average hides these differences. Classroom evaluation needs to examine Filipino and English performance separately.
Types of transcription error
Two models with similar WER can make different errors. In a reading assessment, words omitted from a transcript can be mistaken for words the child skipped.
| Model | Words invented | Words misheard | Words skipped | Share of passage returned |
|---|---|---|---|---|
tiny | 6.8 | 45.5 | 29.3 | 0.79 ⚠️ |
base | 9.1 | 43.2 | 12.1 | 0.99 |
small | 4.9 | 25.0 | 16.2 | 0.91 |
medium | 3.9 | 19.2 | 7.4 | 0.97 |
large-v3-turbo | 4.6 | 14.0 | 26.7 | 0.79 ⚠️ |
large-v3 | 6.4 | 14.5 | 13.6 | 0.94 |
The words skipped column needs particular care. A short transcript can omit words the child read, lowering the score. This analysis treats a returned share below ~0.85 as a warning that requires review before assessment use.
What this means for the app
- Select a model using measurements from the intended server hardware.
- Fit and validate a bias correction for the chosen model. A correction from another model may not transfer.
- Check the Filipino column before approving any tier for real classroom use.
What this cost to run
Whisper was used with its pre-trained weights. These timings cover running each of the six models over the recordings. The calibration model is trained separately and takes milliseconds on a CPU.
Measured on one laptop with an 8 GB RTX 4060, over 215 recordings per model.
| Model | Listening | Aligning words | Total GPU time | Typical recording | Average |
|---|---|---|---|---|---|
tiny | 0.14 h | 0.11 h | 0.25 h | 2.6 s | 4.2 s |
base | 0.16 h | 0.10 h | 0.27 h | 3.5 s | 4.5 s |
small | 0.18 h | 1.04 h | 1.22 h | 7.4 s | 20.4 s |
medium | 0.37 h | 0.22 h | 0.59 h | 6.4 s | 9.9 s |
large-v3-turbo | 0.38 h | 0.37 h | 0.76 h | 8.8 s | 12.6 s |
large-v3 | 3.57 h | 0.95 h | 4.52 h | 50.9 s | 75.7 s |
| All six | 4.82 h | 2.79 h | 7.61 h |
On 26 recordings, runaway transcripts took up to ~3 minutes to align instead of ~5 seconds. These cases raise the average processing time. The table reports both typical and average times so deployment planning can account for slow cases. A slow alignment is also a reason to inspect the transcript.
large-v3 accounts for 3.57 h of the 4.82 h spent transcribing,
74% of the total, for one of six models. On an 8 GB card it barely
fits in memory, and the shortage slows down the word-alignment step as well.
The six-model comparison took 7.61 h on this laptop. A deployment uses one model, so its per-recording time is the starting point for capacity planning.
Limitations
- Timings come from one laptop GPU. The server tiers are suggestions about relative cost and need measurements on the proposed hardware.
- WER counts every misheard word equally, including ones a teacher would ignore.
Generated by pipeline/summarize_models.py.