The feedback looks complete. It names the skill being assessed, gives a result and quotes the student's own words. The software accepts the response without a problem.

But the feedback is wrong.

This is the gap between an answer software can read and an answer people can rely on. Structured output means the AI responds in an agreed format, with information in expected places. That makes the response easier to process. It does not make its conclusions correct.

We can see the problem in a fictional English practice tool. Everything in the example is invented, not an account of a real product or incident. The lesson applies anywhere software uses AI responses to make decisions about people's work.

The student got it right. The feedback did not.

The exercise asks:

Write one sentence about two things you did yesterday. Use the past simple.

We are checking one thing: whether the student uses the past simple correctly for two completed actions yesterday. This is the assessment criterion, or the rule the answer is judged against.

The student writes:

Yesterday I went to the market and bought some fruit.

Now imagine the model returns this assessment:

Software can check that “Not demonstrated” is an allowed result. It can also check that the quoted words really appear in the answer. Both checks pass. Neither tells us whether the student actually made a mistake.

The sentence meets our rule. “Went” is the past simple of “go”; “bought” is the past simple of “buy.” These are irregular forms, so they do not end in “-ed.” The British Council's irregular verb reference lists both. The feedback asks the learner to fix something that is already correct.

Finding the student's words is one job. Deciding what those words show is another.

The format is valid and the quote is real. The conclusion is still wrong.

A completed form is not a correct answer

A structured response works much like a completed form. Each piece of information has a place: the result goes here, the quote goes there, and the feedback goes below it.

The rules for that form are called a schema. They can require certain fields, limit the available choices and set rules for numbers. Some schemas can also make one requirement depend on another. But describing a field as “correct feedback” does not give the software a way to check its truth. See the JSON Schema validation specification.

Our tool might allow three results: demonstrated, not demonstrated and uncertain. This stops the model returning an unexpected label that the software cannot handle. It does not tell us which of the three results the student deserves.

Format checks are useful. We just need to be clear about what they check.

OpenAI makes the same distinction in its Structured Outputs announcement: a response can follow the required structure and still contain mistakes.

Once the format passes, we still have three questions: did the feedback use the right answer, is its judgment justified, and what should it be allowed to change?

First, check that it used the right answer

Imagine the student first writes “Yesterday I go to the market.” While the AI is checking it, they change “go” to “went.” Feedback about the first version could now flag a mistake the student has already fixed.

The application should keep a copy of the answer and assessment rules sent for checking. A set of assessment rules is often called a rubric. If either the answer or the rules change, the tool should check again or clearly say that the feedback refers to an earlier version.

The application needs its own record of this link. Asking the AI to say which version it checked is not enough on its own.

Then check the quotation. Does it come from that answer, and does it preserve the student's wording?

One approach asks the AI to identify where a quote starts and ends. The application then takes the words directly from its saved copy. This helps ensure the displayed quote contains the student's actual words, rather than an invented version.

This follows the broader principle of checking AI-generated references against their sources, recommended in NIST's Generative AI Profile, action MS-2.5-003. Here, the source is the student's answer.

We now know where the words came from. We still do not know whether they support the feedback.

Next, check whether the judgment makes sense

In our original example, the AI quotes “went” and “bought” but says the student has not used the past simple. We need to compare its conclusion with the language in the answer and the task we set. Engineers sometimes call this semantic validation: checking whether the meaning makes sense, not just whether the format is allowed.

Some checks have clear rules. We can reject feedback about a skill the exercise did not ask us to assess. We can stop feedback about an old answer from appearing as feedback on a new one.

Others need more judgment. Are the verbs used correctly in this sentence? Did the learner answer the question? Is the feedback pointing out an error or just suggesting a different way to write?

A reliable verb list can confirm that “went” and “bought” are past simple forms. It cannot tell us that every sentence containing them is correct. We still have to read them in context.

Adding a label such as “evidence verified” does not settle this. Verified how, and by what check? If the AI adds that label to its own response, it is making another claim, not proving the first one.

For this tool, I would keep the rule, the relevant text and the reason for each judgment together. That makes the judgment easier to review. A convincing explanation can still be wrong.

Passing one check does not answer the next question.

Context matters especially when the feedback says something is missing. Quoting only “to the market” does not prove that the answer has no past tense verbs. To make that claim, the tool must examine the full answer, not just the selected words.

A rewrite is not necessarily a correction

Suppose the tool suggests replacing “bought some fruit” with “purchased some fruit.” Both fit our past tense exercise. The change may be useful in a vocabulary lesson, but presenting it as a required grammar correction would misrepresent the learner's answer.

The learner needs to know whether the tool is correcting a mistake, offering an optional alternative or assessing a particular skill. More polished wording does not mean the original was wrong.

Research reflects this distinction. Davis and colleagues tested ten language models on four sets of English correction tasks. Results differed depending on whether the tests rewarded small corrections or allowed broader changes to improve fluency. How we define a good edit matters when we measure the tool. See their ACL 2024 study.

A small code example: what this check can tell us

This optional JavaScript example checks three things: the answer and rules have the expected versions, the criterion is known, and the quotation matches the saved answer. It does not decide whether the feedback is correct. If you do not read code, skip to “The important part is the result” below.

export function checkEvidence(proposal, context) {
  if (proposal.answerRevision !== context.answerRevision ||
      proposal.rubricRevision !== context.rubricRevision) {
    return { kind: "context_mismatch" };
  }
  if (!context.criteria.includes(proposal.criterion)) {
    return { kind: "unknown_criterion" };
  }
  const { start, end, quote } = proposal.evidence;
  if (!Number.isInteger(start) || !Number.isInteger(end) ||
      start < 0 || end <= start || end > context.answer.length) {
    return { kind: "invalid_span" };
  }
  if (context.answer.slice(start, end) !== quote) {
    return { kind: "quote_mismatch" };
  }
  return { kind: "source_verified", judgment: "unchecked" };
}

The important part is the result: source verified, judgment unchecked. It means “these are the student's words,” not “this feedback is correct.”

The accompanying test deliberately sends our wrong judgment through this check. It passes, as it should. The quote is real even though the conclusion is wrong.

That is a useful thing to test. The result's name should say exactly what was checked, so another developer does not later mistake it for approval of the whole response.

For implementation: the function assumes a format check has already confirmed the required fields and data types. The application supplies its own saved request details. Quote positions use JavaScript string indices, so both sides must use the same counting convention. This is one limited check, not a complete feedback system.

Can another AI check the first one?

A second AI model could compare the feedback with the task, full answer and assessment rules. It may catch a mistake the first model missed. A teacher could review cases that remain uncertain or could affect an important decision.

Neither guarantees a correct result. Two models can agree and both be wrong. A human can also miss an error. I would show reviewers the student's work before asking them to consider the AI's explanation, so that explanation is not their only starting point.

The reviewer should see the original task, full answer and assessment rule together. The AI's judgment should be clearly marked as a suggestion that can be corrected. Sending something “for review” only helps if someone has the time and authority to act on it.

There is useful research on combining systems for language correction. Park and colleagues studied unnecessary edits and developed a method that follows an AI model's edits with a smaller, specially trained correction model. Their results concern performance on correction tests, not whether the feedback helps students learn. See their EMNLP 2025 paper.

The lesson is not to avoid automation. It is to know what each added check contributes and where it can still fail.

Then decide what the feedback may change

Suppose the judgment is well supported. Should it produce a practice hint, update a grade or mark a skill as completed? Those are different decisions.

A tool could allow a tentative hint but require a teacher's review before changing a recorded grade. The people designing the product need to set those rules. A correctly formatted AI response should not decide its own authority.

This does not make practice feedback harmless. A wrong hint can still confuse a learner, so there should be a clear way to question or correct it. Being suitable for one use does not automatically make the same feedback suitable for another.

UNESCO's guidance calls for checking whether generative AI is educationally appropriate and ethically sound. That goes beyond making its output readable by software. See Guidance for generative AI in education and research.

Permission to offer a hint is not permission to change a grade.

Test for wrong feedback, not just broken responses

Counting how often the AI fills out the form correctly will not tell us how often it helps or misleads a student.

Tests should cover different ways the process can go wrong: a missing field, a quote from an old answer, or a real quote followed by a wrong conclusion. They should also check that even correct feedback cannot change a grade without the required review.

For the language exercise, include correct forms such as “went,” regular forms such as “walked,” errors such as “buyed,” and sentences that mix correct and incorrect forms. Include valid alternative wording and rewrites that change the student's meaning. Language educators should review the expected answers. Where they disagree, record that disagreement instead of assuming the model is wrong.

Correct answers matter just as much as incorrect ones. A tool may catch genuine mistakes while also telling students to change good answers. Keep optional vocabulary suggestions separate from actual corrections when measuring its performance.

For this fictional tool, I would track:

• Wrong answers accepted as correct.
• Correct answers marked as wrong.
• Explanations that do not fit the answer or task.
• How often the tool needs a reviewer.

I would also check which cases the tool handles on its own. Good results mean less if it sends almost every difficult case to a teacher. These are proposed tests, not results measured from a real product.

Repeat these checks when the model, instructions, assessment rules or intended use changes. Getting the correction right is important. Finding out whether the feedback actually helps students learn requires a separate study of learning outcomes.

What to ask before trusting the next response

Before using an AI response, ask:

1. Did it follow the required format?
2. Did it assess the right version of the answer, using the right rules?
3. Are the quoted words really there?
4. Do those words, in context, justify the conclusion?
5. What may this result change, and who can correct it?
6. Have we tested examples where each of these checks could fail?

Our fictional student used the right verbs. The AI quoted them accurately and still reached the wrong conclusion. A format check and a quote check could not catch that mistake because neither was designed to judge the answer.

That is the broader lesson: a response can be easy for software to read and still be wrong about what it has read. Before letting it guide a person or change a record, we need checks that match the decision we are asking it to support.

{
  "action": "billing_queue",
  "confidence": 0.94,
  "evidence": ["The case includes an invoice number and a duplicate charge."]
}

The request succeeded. The response uses valid JSON, a standard data format. It matches the schema. The confidence is high.

None of that proves the case should be routed automatically.

The caller might not be authorized to trigger that action. Automatic routing might be disabled for the account. The cited evidence might not exist in the original case. The action might be unavailable during an incident. The model might have been given stale context. Even the meaning of 0.94 may be less stable than the number suggests.

This is the architectural mistake behind many fragile AI features: treating a successful model response as a completed product decision.

The model is not the system. It is one probabilistic component inside the system.

Here, probabilistic means the model can produce an uncertain or different answer when a request looks similar. Deterministic means the surrounding software follows explicit rules and produces the same decision from the same checked facts.

I find it useful to design a deterministic shell around that component. The model interprets messy material and proposes a result. Conventional software checks its format, evidence and business rules before changing a real record.

In plain terms: the model suggests, the application checks, business rules permit or block, and storage records. Uncertainty and system failure leave through separate paths.

For teams building AI-backed workflows, the useful question is not “How much should we trust the model?” It is “Where does permission to act live?”

The shell does not make the model deterministic. It gives uncertainty somewhere controlled to go.

A valid format is not permission to act (schema and authorization)

A schema is a list of rules for a response: which fields must exist, what each field may contain and which values are allowed.

Structured-output features are valuable because they make a model follow those rules. They replace a large class of brittle parsing code with a declared contract. For example, OpenAI's Structured Outputs can constrain a response to a supplied JSON Schema. That gives an application a stronger guarantee about the shape of a response than free-form text provides. The API documentation describes that guarantee as schema adherence.

That guarantee is about format. In technical terms, the JSON Schema validation specification defines assertions about the structure of instance data. It does not know the application's permission rules or evidence standards.

But a schema can tell us only that a field called action contains one of the allowed strings. It cannot determine whether that action is justified in this particular business context.

Consider these three questions:

1. Is it well formed? Does the response match the expected type and schema?
2. Is it supported? Does the referenced evidence exist, and does it support the proposed interpretation?
3. Is it permitted? May this system perform that action for this user, account and current operating state?

Structured generation helps with the first question. The application still owns the other two.

That distinction is also a security boundary. The Open Worldwide Application Security Project (OWASP) warns that model output must be checked and cleaned before other software uses it. The risk grows if model output can trigger actions that the original user could not. OWASP classifies this as LLM05:2025.

Treating model output as a formal proposal with named fields, not as an instruction, keeps that boundary visible.

Keep suggestions separate from permission to act (interpretation and authority)

Models are useful where inputs are ambiguous and the application needs interpretation:

• extracting facts from unstructured material;
• classifying intent;
• comparing a document with a rubric;
• proposing a next action;
• generating an explanation for a result already determined elsewhere.

Authority is different. Authority means permission to change the world. It approves, rejects, pays, publishes, deletes, routes, suspends or changes a saved business record, also called durable state.

The model can contribute evidence to that process without owning it.

The application therefore needs three honest kinds of result:

• a proposal supported by evidence;
• an explicit statement that there is not enough evidence;
• a technical failure, such as a timeout or unavailable service.

The second result means the system is uncertain. The third means the system failed to complete the work. Neither should be disguised as a normal business decision.

Optional technical example

The next two TypeScript examples show how software can preserve those three results. They add implementation detail, but they are not required to understand the argument. Readers who do not work with code can skip to What the code guarantees.

A formal result definition, often called a contract, makes the separation explicit:

type ModelProposal =
  | {
      kind: 'proposal';
      action: 'billing_queue' | 'security_queue';
      confidence: number;
      evidence: string[];
    }
  | {
      kind: 'insufficient_evidence';
      missing: string[];
    }
  | {
      kind: 'model_failure';
      reason: 'timeout' | 'invalid_output' | 'unavailable';
      retryable: boolean;
    };

This type makes two decisions before any model is called.

First, uncertainty is legitimate. The model does not have to manufacture a proposal when the input lacks sufficient evidence.

Second, a model failure remains a system failure. It cannot be silently converted into a business result.

The application then applies its business rules, or policy:

function authorizeAction(
  proposal: ModelProposal,
  policy: RoutingPolicy,
): RoutingDecision {
  if (proposal.kind === 'model_failure') {
    return {
      kind: 'system_failure',
      reason: proposal.reason,
      retryable: proposal.retryable,
    };
  }

  if (proposal.kind === 'insufficient_evidence' || proposal.evidence.length === 0) {
    return { kind: 'needs_review', reason: 'insufficient_evidence' };
  }

  if (!policy.automaticRoutingEnabled) {
    return { kind: 'needs_review', reason: 'automatic_routing_disabled' };
  }

  if (!policy.permittedActions.has(proposal.action)) {
    return { kind: 'needs_review', reason: 'action_not_permitted' };
  }

  if (proposal.confidence < policy.minimumConfidence) {
    return { kind: 'needs_review', reason: 'low_confidence' };
  }

  return {
    kind: 'authorized',
    action: proposal.action,
    evidence: proposal.evidence,
    policyVersion: policy.version,
  };
}

What the code guarantees

The important feature is not the syntax. It is the location of authority. The model may propose security_queue, but it cannot grant itself permission to use that route. High confidence cannot enable a feature that business rules have disabled. Missing evidence cannot be repaired by optimism.

The eight layers of the shell

The shell is a way to organize responsibility, not a software library. Its implementation will vary, but eight responsibilities recur in serious systems.

1. Check the request (input contract)

Check the request's size, type, required information, who made it and what that person is allowed to do before sending anything to the model. This is the input contract. An expensive model is not an input-checking service.

2. Record what the model saw (context assembly)

Record the instructions, business records, rule versions and documents given to the model. This is context assembly. Without that record, two apparently identical requests may produce decisions that cannot be explained or reconstructed later.

3. Limit time, cost and retries (invocation budget)

Set explicit limits for waiting time, retry attempts, generated text, simultaneous requests and cost. Engineers call this an invocation budget. “Call the model until it works” is not a failure policy.

4. Check the response format (structural validation)

Require the declared response shape. This is structural validation. Reject unknown result types, missing fields and malformed values. Do not let response-reading software silently invent values for fields that affect a decision.

5. Check whether the claims are supported (semantic validation)

Check the model's claims against material the application can verify. This is semantic validation. Confirm that referenced records exist, quoted evidence is real, numeric ranges make sense and the proposed action fits the current situation.

6. Apply business rules (policy decision)

Apply permissions, account settings, risk limits, enabled features and rules that must always hold. This policy decision happens in ordinary code. It is where a proposal becomes permitted, rejected or sent for review.

7. Save the result safely (durable state transition)

Save important changes so they survive a restart. Engineers call this a durable state transition. Make the save idempotent: repeating the same request or receiving the same event twice must not perform the consequential action twice.

8. Keep a record and monitor the process (audit and observability)

Record the proposal, evidence, rule version, final decision and reason for any failure. Measure each stage separately. This creates an audit record and makes the process observable. A service may appear healthy while document retrieval is broken, the model connection is changing or the business-rule checks reject nearly everything.

This system-level view is consistent with the AI Risk Management Framework from the U.S. National Institute of Standards and Technology (NIST). It treats governance, measurement and management as concerns throughout a system's life. Its Core calls for documented controls around outside AI components, defined human oversight, interpretation of outputs in context and systems that can fail safely beyond their knowledge limits. The framework is broader than model quality alone.

The same boundary appears in the multinational Guidelines for Secure AI System Development. They recommend limiting the actions AI components can trigger, adding independent safety mechanisms, giving each component only the access it needs and checking model outputs before use.

“Needs review” is a real result

Many applications recognize only two results: success and failure.

When a probabilistic component is added, teams often overload one of those states. Low confidence becomes rejection. A timeout becomes “not approved.” Missing evidence becomes an empty result that later parts of the system interpret as negative.

Those mappings are convenient and wrong.

A mature system normally needs at least three outcomes:

• the evidence and business rules permit an action;
• the evidence or business rules require review;
• the system could not produce a trustworthy proposal.

The second path represents uncertainty. The third represents failure. Combining them damages both product behaviour and operations. Review queues become filled with infrastructure incidents, while users receive negative outcomes caused by unavailable dependencies.

Confidence does not solve this. A confidence score can contribute to a review rule, but it is not evidence, permission or proof that the score is reliable. A system should explain why the evidence was sufficient under particular business rules, not merely report that the model sounded certain.

Preserve the identity of failure

Suppose the model call times out. The application has several honest options:

• retry a limited number of times within a fixed time and cost budget;
• place the work in a queue that survives a service restart;
• return a retryable system error;
• continue with reduced but documented capability;
• ask a human to complete the operation.

“Return the negative business result” is not one of them.

Keeping the reason for failure matters because different people and systems can respond. A user may correct a badly formed request. An operator may investigate an outside service outage. A worker may retry a temporary error. A product owner may decide whether reduced operation is acceptable.

Flattening all of these into false or score: 0 destroys that coordination.

Giving each result a clear name keeps the reason intact as it moves through the system. It also creates better measurements: business rejection rate, insufficient-evidence rate and system-failure rate are different signals. Combining them produces a number that tells nobody what to fix.

Test the surrounding system without relying on a live model

The deterministic shell creates a large surface that can be tested without a live model. Small automated checks, commonly called unit tests, can prove that the application's rules behave as intended.

Those tests can prove that:

• an action without evidence never becomes authorized;
• a disabled capability cannot be enabled by high confidence;
• a disallowed action goes to review;
• timeouts remain technical failures and state whether retrying may help;
• repeated versions of the same operation cannot produce duplicate changes;
• every saved decision records the version of the business rules used.

Integration checks can verify the request and response formats against a controlled stand-in for the model service. Engineers often call these adapter contract tests. A small, controlled set of live checks can then detect whether the real service still accepts the schema and returns compatible responses.

That division is useful because live model tests answer a different question. They can reveal a broken connection or provider drift, meaning changes in behaviour over time. They should not be the only proof that the application's essential rules always hold.

The shell has limits

A deterministic shell does not make an AI system automatically correct merely because it was designed this way.

Checks of meaning and evidence, known as semantic validation, can be incomplete. Evidence extraction can miss relevant facts. A business rule can encode the wrong decision. Human review can become a slow queue that nobody monitors. A model can be systematically wrong while still producing perfectly formed proposals.

The point is narrower: the architecture should reveal where each responsibility lives.

The model owns uncertain interpretation. Validation owns format and evidence checks. Policy owns permission to act. Storage owns the saved business record. Operations owns recovery and monitoring. Humans own the decisions the organization has chosen not to automate.

Those boundaries make failure inspectable. They make consequential actions harder to trigger accidentally. They allow individual components to improve without silently changing the meaning of the entire product.

Eight questions for an architecture review

Use these questions before allowing a model response to change a saved business record.

1. What is the model allowed to propose?
2. What evidence must accompany that proposal?
3. Which checks establish that the evidence is real and relevant?
4. Where does the application decide whether the action is permitted, known as authorization?
5. Which explicit result represents insufficient evidence?
6. Can a timeout or malformed response become a user judgment?
7. Can the same change be retried safely without happening twice?
8. Can the decision be reconstructed from its evidence, what the model saw and the version of the business rules used?

If the answer to several of these questions is “inside the prompt,” the system probably has an authority problem.

The hard part of production AI is rarely obtaining a response. It is deciding what that response is allowed to mean.