Research behind READSmart
We study how speech recognition can help teachers assess oral reading. The current findings come from 167 recordings by 42 pupils at one school. They need testing in other schools before we can apply them more widely.
Study questions
We compare transcription errors and processing time across speech-recognition models, test whether calibration reduces measurement bias, and examine differences by grade and language.
What calibration changed
Calibration reduced the average offset in words correct per minute (WCPM) between automated and teacher scores, in grouped cross-validation of 167 recordings from 42 pupils.
−13.603
mean bias, before calibration (WCPM)
0.149
after calibration (WCPM)
Individual scores still differ from a teacher’s. Average absolute error is 10.043 WCPM. The limits of agreement narrowed from a width of 69.605 to 60.434 WCPM, but were still −30.068 to +30.366 WCPM, about 3.2× wider than the difference between two teachers (±9.553).
The teacher reviews the measurements and retains final authority.
How we measured thisSelected findings
- Finding 1: Agreement between teachersTeachers agreed strongly on words read correctly, while agreement on miscues and reading accuracy was less certain.
- Finding 2: Comparing Whisper model sizesThe medium model had the lowest word error rate in this sample; we also measured processing time to compare its hardware cost with the other models.
- Finding 3: Calibrating scores against teacher measurementsCalibration reduced the average reading-speed offset against teachers from -13.603 to 0.149 WCPM in student-grouped cross-validation, while individual errors remained substantial.