Finding 01
Finding 1: Agreement between teachers
In one sentence: Teachers agreed strongly on words read correctly, while agreement on miscues and reading accuracy was less certain.
Comparing teacher scores
We compare the system's scores with teacher scores. To measure uncertainty in that reference, more than one teacher scored some of the same recordings.
What was measured
- 204 scoring sheets in total, covering 167 recordings
- 25 of those recordings were scored by two or more different teachers
- 5 teachers took part
- 18 recordings were scored by 2 teachers; 7 were scored by 3 teachers
Agreement by measure
The intraclass correlation coefficient (ICC) compares variation between students with variation between teachers' scores. An ICC of 1.00 means perfect agreement; this analysis uses 0.75 as the threshold for good agreement.
Only 25 recordings had multiple scorers, so each estimate includes a 95% confidence interval. An interval spanning poor and good agreement leaves the result uncertain.
| What was scored | Agreement (ICC) | How sure (95% range) | Verdict | Two teachers usually within |
|---|---|---|---|---|
| Words read correctly | 0.980 | 0.95 – 0.99 | excellent | ±12.029 words |
| Reading time | 1.000 | 1.00 – 1.00 | excellent | ±0.0 seconds |
| Total miscues | 0.588 | 0.14 – 0.77 | too uncertain to call | ±11.409 miscues |
| Word reading accuracy | 0.427 | -0.09 – 0.71 | too uncertain to call | ±11.734 % |
| Words Correct Per Minute (derived) | 0.960 | 0.90 – 0.98 | excellent | ±9.553 WCPM |
The last column estimates how far apart two teachers' scores fall 95% of the time.
⚠️ Reading time has zero variation between scorers. This suggests that scorers shared a recorded time. It cannot establish independent agreement on timing.
⚠️ Words Correct Per Minute is calculated from other fields on the sheet. Its high agreement depends on those inputs and provides no separate confirmation of teacher agreement.
Reading ICC alongside absolute differences
ICC is a ratio. When students have similar scores, ICC can be low even if teachers give them nearly the same scores. The table below shows the spread between teachers and between students separately.
| What was scored | How far apart the teachers were | How far apart the students were | Reading |
|---|---|---|---|
| Words read correctly | ±3.154 words | ±31.094 words | greater variation between students than between teachers |
| Reading time | ±0.0 seconds | ±42.814 seconds | no timing variation between scorers; see shared-time caveat above |
| Total miscues | ±2.859 miscues | ±5.912 miscues | greater variation between students than between teachers |
| Word reading accuracy | ±2.624 % | ±4.786 % | students are similar, so ICC understates agreement |
| Words Correct Per Minute | ±2.439 WCPM | ±17.593 WCPM | greater variation between students than between teachers |
Whether the absolute differences affect an assessment needs to be considered alongside ICC.
Reading level (Independent / Instructional / Frustration)
- Teachers picked the same level 40.0% of the time (25 recordings).
- Kappa, which adjusts agreement for chance, was 0.133.
Reading level is a category, not a number, so it uses percent agreement and kappa rather than ICC.
Disagreements near level cutoffs
Phil-IRI uses percentage cutoffs: 97% and above is Independent, while below 90% is Frustration. Scores of 96.8% and 97.1% fall into different categories despite being close together.
- Teachers disagreed on the level for 15 recordings.
- All 15 had a score within 3 points of a cutoff.
These disagreements occurred near the category boundaries. Showing the teacher-entered percentage alongside the level would make borderline decisions visible during review.
Differences by teacher
Positive values mean a teacher gave higher scores than colleagues on the same recording; negative values mean lower scores. Each estimate is based on a small number of shared recordings.
Words read correctly
| Teacher | Difference vs. colleagues | Based on |
|---|---|---|
| T01 | +0.9 words | 5 shared recordings |
| T02 | +3.111 words | 9 shared recordings |
| T03 | -5.367 words | 15 shared recordings |
| T04 | +0.417 words | 12 shared recordings |
| T05 | +2.688 words | 16 shared recordings |
Reading time
| Teacher | Difference vs. colleagues | Based on |
|---|---|---|
| T01 | 0.0 seconds | 6 shared recordings |
| T02 | 0.0 seconds | 9 shared recordings |
| T03 | 0.0 seconds | 16 shared recordings |
| T04 | 0.0 seconds | 12 shared recordings |
| T05 | 0.0 seconds | 16 shared recordings |
Total miscues
| Teacher | Difference vs. colleagues | Based on |
|---|---|---|
| T01 | -1.25 miscues | 6 shared recordings |
| T02 | -2.778 miscues | 9 shared recordings |
| T03 | +4.188 miscues | 16 shared recordings |
| T04 | +0.917 miscues | 12 shared recordings |
| T05 | -2.844 miscues | 16 shared recordings |
Word reading accuracy
| Teacher | Difference vs. colleagues | Based on |
|---|---|---|
| T01 | +1.083 % | 6 shared recordings |
| T02 | +3.0 % | 9 shared recordings |
| T03 | -5.1 % | 15 shared recordings |
| T04 | +0.8 % | 10 shared recordings |
| T05 | +2.333 % | 15 shared recordings |
Words Correct Per Minute
| Teacher | Difference vs. colleagues | Based on |
|---|---|---|
| T01 | +0.732 WCPM | 6 shared recordings |
| T02 | +2.197 WCPM | 9 shared recordings |
| T03 | -3.966 WCPM | 16 shared recordings |
| T04 | +0.678 WCPM | 12 shared recordings |
| T05 | +1.948 WCPM | 16 shared recordings |
Implications for the study
- Keep teacher scores unchanged. There are too few shared recordings to estimate reliable corrections for individual teachers.
- Use WCPM as the primary agreement measure, while noting that it is derived from word count and recorded time.
- Report uncertainty in accuracy and miscue agreement, including the effect of similar student scores on ICC.
- Report the underlying teacher-entered percentage alongside the reading level because category assignments can change near a cutoff.
The human reference
Teachers independently judged words read correctly, with strong agreement in this sample. WCPM is calculated from that count and the recorded time.
We compare the system's disagreement with a teacher against the disagreement between two teachers. For words read correctly, the estimated teacher-to-teacher band is ±12.029 words. This finding establishes the comparison band; the system's scores must be tested against it separately.
Limitations
- Only 25 recordings had multiple scorers, which limits the precision of these estimates.
- Teachers were not assigned at random. Each recording was scored by whoever was available, so the analysis uses a one-way (ICC1) reliability model rather than a fully crossed design.
- Per-teacher differences are based on only a handful of shared recordings each and should not be used to judge any individual teacher.
- Teacher names are replaced with T01, T02, ... in this file.
Generated by pipeline/teacher_agreement.py. Regenerate with uv run python teacher_agreement.py.