Featured photograph: Document discussion. Photo: Ketut Subiyanto, Pexels License. Full frame, display-scaled only. Contextual stock photograph, not the fictional assessors, learner responses or scores.
The same words can produce different scores
Two assessors can both read a rubric carefully and still use “relevant limitation” differently. One accepts “we need more tests.” Another expects the response to name a feature of the evidence that limits the conclusion. If that boundary remains unspoken, a polished rubric can conceal two scoring rules.
Calibration makes those interpretations visible before a full marking run. Give each assessor the same small set of responses, ask for independent criterion scores and evidence notes, then discuss the differences. The aim is not to make one person’s total prevail. It is to decide what observable evidence belongs at each level and record the boundary for later work.
Carnegie Mellon’s guidance on rubrics distinguishes criteria, descriptors and performance levels and identifies consistency across graders as a purpose. The three-response routine here is an original practical example, not a CMU protocol or a reliability study. Agreement on three anchors does not establish that a rubric will be used consistently across a cohort.
Use one prompt and a shared evidence packet
All data, responses and assessor disagreements below are fictional. The task is to write a short conclusion about paper-helicopter fall time: “Compare the two designs in this test. Use numerical evidence, explain what the comparison supports, and identify one feature of the investigation that limits the conclusion.” This is a written reasoning example, not instructions to conduct a physical drop activity.
The supplied record says that one short-wing prototype and one long-wing prototype were each dropped three times from the same 2-meter height using the same timing procedure. Short-wing times: 1.1, 1.3 and 1.2 seconds. Long-wing times: 1.8, 2.0 and 1.9 seconds. The same paper type was used. No additional prototypes were tested, and other construction differences were not measured.
The means are 3.6 ÷ 3 = 1.2 seconds and 5.7 ÷ 3 = 1.9 seconds. The long-wing prototype took 0.7 seconds longer on average. For the same distance, a longer fall time means a lower average downward speed. These records describe two prototypes under these conditions; they do not isolate wing length from every other possible difference in their construction.
State the starting rubric before scoring
Use three separate criteria, each scored 0–2. Claim: 2 = accurately compares fall time and limits the claim to this test; 1 = gives an accurate direction but leaves the scope general; 0 = gives no interpretable or supported comparison. Evidence: 2 = accurately uses numerical results for both designs; 1 = uses a relevant accurate result for only one design; 0 = supplies no accurate relevant numerical evidence. Equivalent accurate comparisons, including the full trial values, are acceptable; the means are not the only permissible evidence.
Reasoning: 2 = explains how the time comparison supports the conclusion and notes a relevant limitation; 1 = does one of those two things; 0 = does neither. That last descriptor deliberately contains a boundary to examine: what will count as “notes a relevant limitation”? The learner prompt already asks for a limiting feature of the investigation. The calibration should make that expectation usable, not introduce an unrelated demand after seeing the responses.
Three anchors, with the reasoning left visible
Anchor A: a general claim and one result
“Long-wing helicopters fall more slowly. The long-wing average was 1.9 seconds.” A proposed score is Claim 1, Evidence 1, Reasoning 0. The direction matches the supplied comparison, but the claim extends to long-wing helicopters generally. The response gives only one design’s numerical result and does not explain the comparison or name a limitation. Do not infer the missing short-wing result from what the writer might have intended.
Anchor B: a comparison, plus a vague next step
“In these tests, the long-wing prototype fell more slowly: its mean time was 1.9 seconds against 1.2 seconds for the short-wing prototype. The extra 0.7 seconds over the same distance supports that comparison. We need more tests.” Claim 2 and Evidence 2 are defensible under the starting rubric. The Reasoning score is where the assessors disagree.
Anchor C: a comparison with a named limit
“In these tests, the long-wing prototype fell more slowly: its mean time was 1.9 seconds against 1.2 seconds for the short-wing prototype. The extra 0.7 seconds over the same distance supports that comparison. Only one prototype of each design was tested, so other differences in how those two models were made could explain part of the difference. Test additional prototypes before generalizing.” Proposed scores: Claim 2, Evidence 2, Reasoning 2. The limitation is connected to this investigation and explains why the conclusion should remain bounded.
Figure 1. Compare evidence in the anchors, not just their overall rank. Original fictional excerpts and annotations by The Instruction Edit.
Score independently, then compare one criterion at a time
For each anchor, each assessor records three scores and copies the words that justify each one. Do this before discussing the responses. A shared total alone can hide a disagreement: one assessor could award more for evidence and less for reasoning, while the other does the reverse. Keep the criterion scores visible.
In this fictional calibration, Assessor 1 scores Anchor B as 2 / 2 / 2. They treat “We need more tests” as a limitation because it acknowledges uncertainty. Assessor 2 scores it 2 / 2 / 1. They see an explained comparison, but no named feature that limits it. Both point to the same sentence. The disagreement is not about the arithmetic; it is about the boundary in the descriptor.
Figure 2. Criterion-level comparison isolates the descriptor disagreement. Original hypothetical scoring example by The Instruction Edit; not observed assessor data.
Ask each assessor which exact phrase satisfied the descriptor. Then ask what distinguishes B from C. C identifies the one-prototype-per-design problem and connects it to possible construction differences. B recommends a next step without saying what the present evidence cannot resolve. If the task is meant to assess identification of a limiting feature, that difference should appear in the decision rule.
Record a clarification, not just an agreed total
For this task, the clarified Reasoning level 2 is: “Explains how the comparison supports the conclusion and names a feature of the supplied investigation that limits it, with a link to what remains uncertain.” Level 1 is met by the explained comparison alone, or by a relevant named limitation with its uncertainty link alone. Level 0 meets neither. Under that clarification, B receives Reasoning 1 and C receives Reasoning 2. “More tests” by itself is a next step, not the missing link.
Figure 3. Preserve a clarified rule and examples on each side. Original descriptor-boundary record by The Instruction Edit.
Record the date, criterion, disputed phrase, decision and anchors that illustrate it. For this fictional set: “Reasoning boundary: a generic request for more testing does not identify a limiting feature. B = 1; C = 2. Accept other relevant limitations if they identify a feature in the packet and explain the uncertainty.” Do not turn C’s exact wording into a password. A learner can meet the criterion in different language.
If calibration reveals that the intended rule changes what learners were asked to do, revise the prompt and rubric before administering the assessment. If work has already been submitted, follow the institution’s moderation and fairness procedures rather than silently adding a new requirement. This example is a pre-administration rehearsal, not a recommendation to change standards retrospectively.
Check a fresh response before the full marking run
After recording the boundary, score another response independently. A fresh case tests whether the decision can be applied beyond the three discussed anchors; it is not proof of reliability. During marking, keep a short query log for new edge cases and periodically compare a small common sample. Escalate unresolved interpretations through the agreed marking process rather than inventing a private exception.
Use fictional or appropriately anonymized anchors, with permission and local procedures for any real learner work. The document-discussion photo is contextual stock photography, not a record of these assessors, students or scores. The original anchor pack and disagreement diagram show the hypothetical reasoning in full.
Before committing to a full set of scripts, independently score three anchors with a colleague. Leave the meeting with one clarified boundary, an evidence example on each side of it, and a record of what has not yet been settled.
Source and scope
Creating and Using Rubrics — Carnegie Mellon University Eberly Center. Official guidance, checked October 30, 2026. The rubric, fictional investigation, responses, scores and calibration log are original examples. No student work, observed assessor agreement, classroom trial or measured reliability gain is claimed.
