Structured Output Is Not Trusted Output
What an AI language tutor must check before correcting a learner
A student writes a correct sentence. An AI tutor tells them to fix it.
The feedback looks complete. It names the skill being assessed, gives a result and quotes the student's own words. The software accepts the response without a problem.
But the feedback is wrong.
This is the gap between an answer software can read and an answer people can rely on. Structured output means the AI responds in an agreed format, with information in expected places. That makes the response easier to process. It does not make its conclusions correct.
We can see the problem in a fictional English practice tool. Everything in the example is invented, not an account of a real product or incident. The lesson applies anywhere software uses AI responses to make decisions about people's work.
The student got it right. The feedback did not.
The exercise asks:
Write one sentence about two things you did yesterday. Use the past simple.
We are checking one thing: whether the student uses the past simple correctly for two completed actions yesterday. This is the assessment criterion, or the rule the answer is judged against.
The student writes:
Yesterday I went to the market and bought some fruit.
Now imagine the model returns this assessment:
| Field | Proposed value |
|---|---|
| Criterion | Use past simple for two completed actions |
| Verdict | Not demonstrated |
| Evidence | “went to the market and bought some fruit” |
| Feedback | “Use past tense verbs to describe what you did yesterday.” |
Software can check that “Not demonstrated” is an allowed result. It can also check that the quoted words really appear in the answer. Both checks pass. Neither tells us whether the student actually made a mistake.
The sentence meets our rule. “Went” is the past simple of “go”; “bought” is the past simple of “buy.” These are irregular forms, so they do not end in “-ed.” The British Council's irregular verb reference lists both. The feedback asks the learner to fix something that is already correct.
Finding the student's words is one job. Deciding what those words show is another.
The format is valid and the quote is real. The conclusion is still wrong.
A completed form is not a correct answer
A structured response works much like a completed form. Each piece of information has a place: the result goes here, the quote goes there, and the feedback goes below it.
The rules for that form are called a schema. They can require certain fields, limit the available choices and set rules for numbers. Some schemas can also make one requirement depend on another. But describing a field as “correct feedback” does not give the software a way to check its truth. See the JSON Schema validation specification.
Our tool might allow three results: demonstrated, not demonstrated and uncertain. This stops the model returning an unexpected label that the software cannot handle. It does not tell us which of the three results the student deserves.
Format checks are useful. We just need to be clear about what they check.
OpenAI makes the same distinction in its Structured Outputs announcement: a response can follow the required structure and still contain mistakes.
Once the format passes, we still have three questions: did the feedback use the right answer, is its judgment justified, and what should it be allowed to change?
First, check that it used the right answer
Imagine the student first writes “Yesterday I go to the market.” While the AI is checking it, they change “go” to “went.” Feedback about the first version could now flag a mistake the student has already fixed.
The application should keep a copy of the answer and assessment rules sent for checking. A set of assessment rules is often called a rubric. If either the answer or the rules change, the tool should check again or clearly say that the feedback refers to an earlier version.
The application needs its own record of this link. Asking the AI to say which version it checked is not enough on its own.
Then check the quotation. Does it come from that answer, and does it preserve the student's wording?
One approach asks the AI to identify where a quote starts and ends. The application then takes the words directly from its saved copy. This helps ensure the displayed quote contains the student's actual words, rather than an invented version.
This follows the broader principle of checking AI-generated references against their sources, recommended in NIST's Generative AI Profile, action MS-2.5-003. Here, the source is the student's answer.
We now know where the words came from. We still do not know whether they support the feedback.
Next, check whether the judgment makes sense
In our original example, the AI quotes “went” and “bought” but says the student has not used the past simple. We need to compare its conclusion with the language in the answer and the task we set. Engineers sometimes call this semantic validation: checking whether the meaning makes sense, not just whether the format is allowed.
Some checks have clear rules. We can reject feedback about a skill the exercise did not ask us to assess. We can stop feedback about an old answer from appearing as feedback on a new one.
Others need more judgment. Are the verbs used correctly in this sentence? Did the learner answer the question? Is the feedback pointing out an error or just suggesting a different way to write?
A reliable verb list can confirm that “went” and “bought” are past simple forms. It cannot tell us that every sentence containing them is correct. We still have to read them in context.
Adding a label such as “evidence verified” does not settle this. Verified how, and by what check? If the AI adds that label to its own response, it is making another claim, not proving the first one.
For this tool, I would keep the rule, the relevant text and the reason for each judgment together. That makes the judgment easier to review. A convincing explanation can still be wrong.
Passing one check does not answer the next question.
Context matters especially when the feedback says something is missing. Quoting only “to the market” does not prove that the answer has no past tense verbs. To make that claim, the tool must examine the full answer, not just the selected words.
A rewrite is not necessarily a correction
Suppose the tool suggests replacing “bought some fruit” with “purchased some fruit.” Both fit our past tense exercise. The change may be useful in a vocabulary lesson, but presenting it as a required grammar correction would misrepresent the learner's answer.
The learner needs to know whether the tool is correcting a mistake, offering an optional alternative or assessing a particular skill. More polished wording does not mean the original was wrong.
Research reflects this distinction. Davis and colleagues tested ten language models on four sets of English correction tasks. Results differed depending on whether the tests rewarded small corrections or allowed broader changes to improve fluency. How we define a good edit matters when we measure the tool. See their ACL 2024 study.
A small code example: what this check can tell us
This optional JavaScript example checks three things: the answer and rules have the expected versions, the criterion is known, and the quotation matches the saved answer. It does not decide whether the feedback is correct. If you do not read code, skip to “The important part is the result” below.
export function checkEvidence(proposal, context) {
if (proposal.answerRevision !== context.answerRevision ||
proposal.rubricRevision !== context.rubricRevision) {
return { kind: "context_mismatch" };
}
if (!context.criteria.includes(proposal.criterion)) {
return { kind: "unknown_criterion" };
}
const { start, end, quote } = proposal.evidence;
if (!Number.isInteger(start) || !Number.isInteger(end) ||
start < 0 || end <= start || end > context.answer.length) {
return { kind: "invalid_span" };
}
if (context.answer.slice(start, end) !== quote) {
return { kind: "quote_mismatch" };
}
return { kind: "source_verified", judgment: "unchecked" };
}
The important part is the result: source verified, judgment unchecked. It means “these are the student's words,” not “this feedback is correct.”
The accompanying test deliberately sends our wrong judgment through this check. It passes, as it should. The quote is real even though the conclusion is wrong.
That is a useful thing to test. The result's name should say exactly what was checked, so another developer does not later mistake it for approval of the whole response.
For implementation: the function assumes a format check has already confirmed the required fields and data types. The application supplies its own saved request details. Quote positions use JavaScript string indices, so both sides must use the same counting convention. This is one limited check, not a complete feedback system.
Can another AI check the first one?
A second AI model could compare the feedback with the task, full answer and assessment rules. It may catch a mistake the first model missed. A teacher could review cases that remain uncertain or could affect an important decision.
Neither guarantees a correct result. Two models can agree and both be wrong. A human can also miss an error. I would show reviewers the student's work before asking them to consider the AI's explanation, so that explanation is not their only starting point.
The reviewer should see the original task, full answer and assessment rule together. The AI's judgment should be clearly marked as a suggestion that can be corrected. Sending something “for review” only helps if someone has the time and authority to act on it.
There is useful research on combining systems for language correction. Park and colleagues studied unnecessary edits and developed a method that follows an AI model's edits with a smaller, specially trained correction model. Their results concern performance on correction tests, not whether the feedback helps students learn. See their EMNLP 2025 paper.
The lesson is not to avoid automation. It is to know what each added check contributes and where it can still fail.
Then decide what the feedback may change
Suppose the judgment is well supported. Should it produce a practice hint, update a grade or mark a skill as completed? Those are different decisions.
A tool could allow a tentative hint but require a teacher's review before changing a recorded grade. The people designing the product need to set those rules. A correctly formatted AI response should not decide its own authority.
This does not make practice feedback harmless. A wrong hint can still confuse a learner, so there should be a clear way to question or correct it. Being suitable for one use does not automatically make the same feedback suitable for another.
UNESCO's guidance calls for checking whether generative AI is educationally appropriate and ethically sound. That goes beyond making its output readable by software. See Guidance for generative AI in education and research.
Permission to offer a hint is not permission to change a grade.
Test for wrong feedback, not just broken responses
Counting how often the AI fills out the form correctly will not tell us how often it helps or misleads a student.
Tests should cover different ways the process can go wrong: a missing field, a quote from an old answer, or a real quote followed by a wrong conclusion. They should also check that even correct feedback cannot change a grade without the required review.
For the language exercise, include correct forms such as “went,” regular forms such as “walked,” errors such as “buyed,” and sentences that mix correct and incorrect forms. Include valid alternative wording and rewrites that change the student's meaning. Language educators should review the expected answers. Where they disagree, record that disagreement instead of assuming the model is wrong.
Correct answers matter just as much as incorrect ones. A tool may catch genuine mistakes while also telling students to change good answers. Keep optional vocabulary suggestions separate from actual corrections when measuring its performance.
For this fictional tool, I would track:
- Wrong answers accepted as correct.
- Correct answers marked as wrong.
- Explanations that do not fit the answer or task.
- How often the tool needs a reviewer.
I would also check which cases the tool handles on its own. Good results mean less if it sends almost every difficult case to a teacher. These are proposed tests, not results measured from a real product.
Repeat these checks when the model, instructions, assessment rules or intended use changes. Getting the correction right is important. Finding out whether the feedback actually helps students learn requires a separate study of learning outcomes.
What to ask before trusting the next response
Before using an AI response, ask:
- Did it follow the required format?
- Did it assess the right version of the answer, using the right rules?
- Are the quoted words really there?
- Do those words, in context, justify the conclusion?
- What may this result change, and who can correct it?
- Have we tested examples where each of these checks could fail?
Our fictional student used the right verbs. The AI quoted them accurately and still reached the wrong conclusion. A format check and a quote check could not catch that mistake because neither was designed to judge the answer.
That is the broader lesson: a response can be easy for software to read and still be wrong about what it has read. Before letting it guide a person or change a record, we need checks that match the decision we are asking it to support.
