Finding 7: A spelling difference can look like a reading error

In one sentence: An approved name spelling such as “Jackie” for “Jacky” should not lower the automated reading estimate solely because its spelling differs; the correction fixes that regression, while rescoring all 1,290 saved model results changed none because the study corpus contains different passages.

What was observed

On 21 September 2026, a user reported that a reading of “Jacky” in Get Up, Jacky! appeared as “Jackie” in the transcript. The comparison treated different spellings as substitutions. Both spellings can represent the same spoken name, so this text difference alone does not establish a pupil reading error.

The report was reproduced with constructed text passed through the scorer. The reported audio was not reviewed in this check. This is a scoring-method correction, not evidence that Whisper heard every reported occurrence correctly or that word-level pronunciation assessment has been validated.

The correction

The shared, reviewed rule accepts Jackie when the reference word is Jacky, only for the complete normalized text of the English Grade 4 passage Get Up, Jacky! (philiri-2018-pre-a-g4-en). A SHA-256 digest of the full normalized passage identifies the rule without putting official assessment text in public browser bundles. A changed passage must be reviewed again. The rule is recorded in pipeline/spelling-variants.json under scoring version oral-reading-v2-spelling-variants.

The Python score and web highlights apply the same rule. Accepted variants are shown in a note, while the original passage and transcript remain visible. Unlisted spellings, other names, missing words, added words, and other passages still use exact word comparison. No general fuzzy spelling or phonetic matching was introduced. Uncertain differences still need teacher review of the audio.

The full-passage check also exposed a repetition-filter interaction: after “Jackie” failed an exact match, the reference cursor stalled and a later valid repeated “Beep!” was removed. The cursor now recognizes the approved variant without rewriting the transcript, preserving both “Ring!” tokens and both “Beep!” tokens.

What the regression check establishes

The constructed transcript copies the entire passage and changes all five instances of Jacky to Jackie. It contains 67 tokens under the scorer's normalization. For illustration, its duration is set to 60 seconds; this is not a measured pupil recording.

Comparison	Substitutions	Omissions	Correct tokens	WCPM at the illustrative duration	Accuracy
Strict spelling comparison, after the shared repetition preprocessing	5	0	62	62	92.54%
Reviewed spelling comparison	0	0	67	67	100%

Strict WER remains 5 / 67 = 0.0746. The separately labeled adjusted WER is 0. The earlier scorer also introduced one false omission through the repetition-filter interaction; that is a software defect, not an additional pupil error.

Regression checks cover preservation of the original transcript, the same counts in Python and TypeScript, real substitutions/omissions/insertions, unknown names, edited passages, strict highlights for older results, and storage of the audit metadata. Reproduce the core checks with python pipeline/tests/test_score.py and pnpm exec vitest run src/lib/miscues.test.ts src/components/passage-comparison.test.tsx.

Rescoring the saved corpus

On 21 September 2026, all 1,290 saved model results were recomputed from their original saved transcripts, word timings, and VAD durations. The audit compared the previous scorer with the corrected scorer and checked the corrected metrics against the saved values. No audio was retranscribed or preamble trimmed again.

Scope	Saved entries	Model results	Changed results
Primary collection	167	1,002	0
Cross-scoring collection	48	288	0
Total	215	1,290	0

Each entry has results for tiny, base, small, medium, large-v3-turbo, and large-v3. The 48 cross-scoring entries are additional ratings/entries, not 48 additional independent pupils or recordings. None of the corpus's 12 reference passages matches Get Up, Jacky!. Consequently, WCPM, accuracy, strict WER, correct-word counts, and miscue counts are unchanged in every saved result, with zero reading-level transitions. Unchanged results were not overwritten.

The live database also had no stored result for the Jacky passage and no retained demo sessions at the time of this check, so no live score required updating. This snapshot does not reconstruct expired demo transcripts.

The model ranking and inputs to the published calibration/agreement analyses are unchanged; Findings 1–6 retain their original results. This is a zero-impact result for the current rule and corpus, not evidence that other name variants never cause errors. Reproduce with python pipeline/experiments/spelling_audit.py --data data --baseline 79618e94fd1507f37998b392387a8088ac4d41a9.

Preserving the research comparison

• wer continues to measure exact spelling differences on the preprocessed transcript. Approved variants do not turn strict WER into adjusted WER.
• raw_wcpm, raw_accuracy, and suggested miscue counts use the reviewed comparison. They remain automated estimates, not teacher-verified Phil-IRI results.
• scoring_audit in offline results retains the rule version, passage key, strict WCPM, strict accuracy, strict WER, strict miscue counts, and separately labeled adjusted WER. The model-sweep detail CSV retains this audit as JSON.
• Worker callbacks carry the same audit as scoringAudit. It is validated and stored with the existing word metadata, subject to the same retention and deletion rules. The classroom result also retains its scoring version after raw-content expiry.
• The original transcript and word timings are not rewritten by the spelling rule. Existing stored results are not automatically rescored, and their highlights keep the earlier policy until they are reprocessed.
• The exported calibration predates this correction. When the correction changes raw WCPM, the runtime withholds that calibrated result rather than silently applying the old fitted model to a changed input measure.

What has not been established

No new ASR run, audio adjudication, calibration refit, or new ICC/Bland-Altman analysis was performed for this correction. The saved corpus was rescored, with no metric changes. The aggregate results in Findings 1–6 remain their original measurements. The constructed example must not be reported as an observed improvement in pupil performance or model accuracy.

Before claiming a research benefit, freeze the reviewed rule list, compare strict and adjusted recording-level measurements against teacher scores on held-out recordings, report English and Filipino separately with grade retained, and assess bias/agreement using Bland-Altman and ICC. Audit whether genuine reading errors are being accepted. Any calibration using adjusted measurements needs fitting and validation under that policy. Word-level highlighting remains assistive and unvalidated.