Finding 03
Finding 3: Calibrating scores against teacher measurements
In one sentence: Calibration reduced the average reading-speed offset against teachers from -13.603 to 0.149 WCPM in student-grouped cross-validation, while individual errors remained substantial.
Correcting measurement bias
The system estimates reading speed from the number of words read correctly and the time taken. Transcription errors can make that estimate consistently too high or too low. Calibration learns a correction from paired automated and teacher measurements. The teacher reviews the corrected measurement before making an instructional decision.
Speech model used: medium, which had the lowest WER in Experiment 1.
Recordings: 167, from 42 students.
Test design
- All recordings from one student stay in either the training or test set for each split. This prevents the same student appearing in both sets.
- We used 5 splits x 3 repeats to reduce dependence on one division of the data.
- Settings were fixed before examining the results.
ridgewas named as the primary method in advance. The remaining methods are exploratory comparisons.
Reading speed (words per minute)
The uncorrected measurement has an average offset of -13.603 WCPM and a mean absolute error of 15.037 WCPM.
| Method | Average miss | Systematic offset | Agreement range | Lower absolute bias? |
|---|---|---|---|---|
| (raw measurement) | 15.037 WCPM | -13.603 WCPM | -48.4 to +21.2 | |
| (guess the average) | 16.513 WCPM | +0.030 WCPM | -45.0 to +45.0 | yes |
linear | 10.038 WCPM | +0.141 WCPM | -30.1 to +30.4 | yes |
ridge ← primary | 10.043 WCPM | +0.149 WCPM | -30.1 to +30.4 | yes |
lasso | 10.045 WCPM | +0.154 WCPM | -30.1 to +30.4 | yes |
elasticnet | 10.092 WCPM | +0.193 WCPM | -30.1 to +30.4 | yes |
random_forest | 11.401 WCPM | +0.950 WCPM | -32.4 to +34.3 | yes |
hist_boosting | 11.532 WCPM | +0.797 WCPM | -32.4 to +34.0 | yes |
svr_rbf | 10.427 WCPM | -0.647 WCPM | -33.8 to +32.5 | yes |
knn | 11.67 WCPM | +1.211 WCPM | -32.0 to +34.4 | yes |
Average miss is the mean absolute error across recordings. Systematic offset is the mean signed error, also called bias. A small offset can coexist with large individual errors when positive and negative errors cancel out.
Differences between recordings
The corrected WCPM estimates had R² = 0.536. They captured some variation between recordings, although the agreement range above still shows substantial individual error.
Word reading accuracy
The uncorrected measurement has an average offset of -19.87 % and a mean absolute error of 19.993 %.
| Method | Average miss | Systematic offset | Agreement range | Lower absolute bias? |
|---|---|---|---|---|
| (raw measurement) | 19.993 % | -19.870 % | -55.2 to +15.5 | |
| (guess the average) | 4.02 % | +0.004 % | -12.3 to +12.3 | yes |
linear | 3.899 % | +0.289 % | -11.8 to +12.3 | yes |
ridge ← primary | 3.896 % | +0.288 % | -11.8 to +12.3 | yes |
lasso | 3.884 % | +0.288 % | -11.8 to +12.4 | yes |
elasticnet | 3.877 % | +0.281 % | -11.7 to +12.3 | yes |
random_forest | 3.963 % | +0.150 % | -11.8 to +12.1 | yes |
hist_boosting | 4.088 % | +0.337 % | -11.7 to +12.4 | yes |
svr_rbf | 3.512 % | +1.177 % | -10.2 to +12.6 | yes |
knn | 3.772 % | +0.857 % | -10.8 to +12.5 | yes |
Limited information about individual accuracy
Corrected accuracy had R² = 0.001, despite a much smaller average offset. The correction provides little information about differences between recordings on this measure. These results do not support ranking pupils or displaying an individual calibrated accuracy estimate.
Primary WCPM result
For reading speed (WCPM), using ridge:
- Systematic offset: -13.603 → 0.149 WCPM (98.9% smaller)
- Mean absolute error: 15.037 → 10.043 WCPM
- Agreement range narrowed from 69.605 to 60.434 WCPM
These estimates come from recordings held out from training, with students kept separate between training and test sets.
Comparison with teacher disagreement
Finding 01 measured the disagreement between teachers scoring the same recording. That gives a reference for interpreting the system's errors.
| Comparison | Agreement within |
|---|---|
| Two teachers, same recording | ±9.6 WCPM |
| System vs teacher, after correction | ±30.2 WCPM |
| System vs teacher, before correction | ±34.8 WCPM |
The system's agreement range is about 3.2× wider than the teacher-to-teacher range. Calibration reduced the average offset substantially, but individual errors remain larger than the differences between teachers. This experiment does not establish how to reduce that remaining spread.
Why the students were kept together
Each child recorded 4-8 passages. Splitting by recording can put the same child in both the training and test sets, making performance on unseen students look better than it is. The table compares the same analysis under the two split methods.
| Measure | Grouped by student (used here) | Grouped by recording (leaky) |
|---|---|---|
| Reading speed R² | 0.536 | 0.546 |
| Reading speed mean absolute error | 10.043 WCPM | 9.844 WCPM |
| Accuracy R² | 0.001 | 0.184 |
| Accuracy mean absolute error | 3.896 % | 3.732 % |
WCPM changes little between the two split methods. Accuracy R² rises from 0.001 to 0.184 when the same student can appear in both sets. We report the student-grouped result to evaluate performance on students absent from training.
Limitations
- The sample has 167 recordings from 42 students. Small differences between methods need validation with more data.
- The primary result uses the method chosen in advance. Comparing ten methods increases the chance that one exploratory method looks best by chance.
- Reading speed here is derived from the teacher's word count and stopwatch time, not judged directly (see Finding 01).
- This corrects a measurement. It does not judge reading ability, and it cannot detect mispronunciation.
Generated by pipeline/calibrate.py.