Finding 06
Finding 6: Exploring groups in the recording measurements
In one sentence: Groups formed from automated recording measurements had only a weak relationship with teacher-assigned Phil-IRI levels in this sample.
Exploring the measurements without teacher labels
This experiment groups recordings using only automated measurements. Teacher scores and reading-level labels are withheld when fitting the groups, then used to compare the grouping with Phil-IRI levels. We report group separation and agreement with teacher labels, without treating the groups as predictions of reading level.
The analysis uses 167 recordings and 20 measurements per recording, from the
medium speech model.
Comparing the number of groups
A silhouette score measures how well groups are separated. This analysis uses scores above 0.5 as a guide to stronger separation and scores below 0.25 as a guide to weak separation. These thresholds do not establish whether groups are real categories of readers.
| Number of groups | Separation score |
|---|---|
| 2 | 0.379 |
| 3 | 0.392 ← highest score |
| 4 | 0.275 |
| 5 | 0.272 |
| 6 | 0.305 |
Three groups had the highest separation score, 0.392, compared with 0.379 for two groups. Both are below 0.50. These measurements provide limited evidence for distinct groups and do not establish natural boundaries between reading levels.
Comparing three groups with Phil-IRI levels
With the number of groups fixed at three, agreement with teacher-assigned levels had an Adjusted Rand Index of 0.071. A value of 0 indicates chance-level agreement and 1 indicates identical groupings.
- Teachers' labels: {'Frustration': 24, 'Instructional': 76, 'Independent': 67}
- Computer's group sizes: [117, 36, 14]
The group sizes also differ: clustering placed 70% of recordings in one group, while the teacher labels were more evenly distributed. Recording quality and extreme fluency values could contribute to this pattern, but the analysis does not establish what caused it.
Overlap between measurements
Many of the 20 measurements vary together. For example, long pauses, a low word rate, and a long recording time can describe related aspects of the same reading.
- 4 underlying dimensions capture 80% of the measured variation
- 8 capture 95%
- The two biggest alone explain 63%
This overlap supports keeping the calibration model small, since several columns contain related information.
Unusual recordings and transcription error
The method flagged 9 recordings as unusual without using teacher labels. Their transcription errors were then compared with those of the remaining recordings.
| Group | Typical recognition error |
|---|---|
| Flagged as unusual | 0.8232 |
| Everything else | 0.2608 |
The flagged recordings had about 3.2× higher recognition error. This suggests that an unusual-recording flag could help identify results needing review. Its use in the app would need separate validation.
Relationship to teacher agreement
In Finding 01, all teacher disagreements about reading level occurred within 3 points of a Phil-IRI cutoff. Here, automated groups matched teacher levels weakly. These results support showing the teacher-entered percentage with the level and making borderline results visible, while leaving the decision with the teacher.
Limitations
- 167 recordings is small for clustering. Treat this as suggestive.
- Only the measurements the ASR pipeline produces were used. A different feature set could group readings differently.
- This clustering analysis does not test the validity of Phil-IRI levels. It describes separation in the selected automated measurements from this sample.
Generated by pipeline/clustering.py.