The Model Is Not the System

A model receives a support case and returns a structured response like this:

{
  "action": "billing_queue",
  "confidence": 0.94,
  "evidence": ["The case includes an invoice number and a duplicate charge."]
}

The request succeeded. The response uses valid JSON, a standard data format. It matches the schema. The confidence is high.

None of that proves the case should be routed automatically.

The caller might not be authorized to trigger that action. Automatic routing might be disabled for the account. The cited evidence might not exist in the original case. The action might be unavailable during an incident. The model might have been given stale context. Even the meaning of 0.94 may be less stable than the number suggests.

This is the architectural mistake behind many fragile AI features: treating a successful model response as a completed product decision.

The model is not the system. It is one probabilistic component inside the system.

Here, probabilistic means the model can produce an uncertain or different answer when a request looks similar. Deterministic means the surrounding software follows explicit rules and produces the same decision from the same checked facts.

I find it useful to design a deterministic shell around that component. The model interprets messy material and proposes a result. Conventional software checks its format, evidence and business rules before changing a real record.

In plain terms: the model suggests, the application checks, business rules permit or block, and storage records. Uncertainty and system failure leave through separate paths.

For teams building AI-backed workflows, the useful question is not “How much should we trust the model?” It is “Where does permission to act live?”

The shell does not make the model deterministic. It gives uncertainty somewhere controlled to go.

A valid format is not permission to act (schema and authorization)

A schema is a list of rules for a response: which fields must exist, what each field may contain and which values are allowed.

Structured-output features are valuable because they make a model follow those rules. They replace a large class of brittle parsing code with a declared contract. For example, OpenAI's Structured Outputs can constrain a response to a supplied JSON Schema. That gives an application a stronger guarantee about the shape of a response than free-form text provides. The API documentation describes that guarantee as schema adherence.

That guarantee is about format. In technical terms, the JSON Schema validation specification defines assertions about the structure of instance data. It does not know the application's permission rules or evidence standards.

But a schema can tell us only that a field called action contains one of the allowed strings. It cannot determine whether that action is justified in this particular business context.

Consider these three questions:

1. Is it well formed? Does the response match the expected type and schema?
2. Is it supported? Does the referenced evidence exist, and does it support the proposed interpretation?
3. Is it permitted? May this system perform that action for this user, account and current operating state?

Structured generation helps with the first question. The application still owns the other two.

That distinction is also a security boundary. The Open Worldwide Application Security Project (OWASP) warns that model output must be checked and cleaned before other software uses it. The risk grows if model output can trigger actions that the original user could not. OWASP classifies this as LLM05:2025.

Treating model output as a formal proposal with named fields, not as an instruction, keeps that boundary visible.

Keep suggestions separate from permission to act (interpretation and authority)

Models are useful where inputs are ambiguous and the application needs interpretation:

• extracting facts from unstructured material;
• classifying intent;
• comparing a document with a rubric;
• proposing a next action;
• generating an explanation for a result already determined elsewhere.

Authority is different. Authority means permission to change the world. It approves, rejects, pays, publishes, deletes, routes, suspends or changes a saved business record, also called durable state.

The model can contribute evidence to that process without owning it.

The application therefore needs three honest kinds of result:

• a proposal supported by evidence;
• an explicit statement that there is not enough evidence;
• a technical failure, such as a timeout or unavailable service.

The second result means the system is uncertain. The third means the system failed to complete the work. Neither should be disguised as a normal business decision.

Optional technical example

The next two TypeScript examples show how software can preserve those three results. They add implementation detail, but they are not required to understand the argument. Readers who do not work with code can skip to What the code guarantees.

A formal result definition, often called a contract, makes the separation explicit:

type ModelProposal =
  | {
      kind: 'proposal';
      action: 'billing_queue' | 'security_queue';
      confidence: number;
      evidence: string[];
    }
  | {
      kind: 'insufficient_evidence';
      missing: string[];
    }
  | {
      kind: 'model_failure';
      reason: 'timeout' | 'invalid_output' | 'unavailable';
      retryable: boolean;
    };

This type makes two decisions before any model is called.

First, uncertainty is legitimate. The model does not have to manufacture a proposal when the input lacks sufficient evidence.

Second, a model failure remains a system failure. It cannot be silently converted into a business result.

The application then applies its business rules, or policy:

function authorizeAction(
  proposal: ModelProposal,
  policy: RoutingPolicy,
): RoutingDecision {
  if (proposal.kind === 'model_failure') {
    return {
      kind: 'system_failure',
      reason: proposal.reason,
      retryable: proposal.retryable,
    };
  }

  if (proposal.kind === 'insufficient_evidence' || proposal.evidence.length === 0) {
    return { kind: 'needs_review', reason: 'insufficient_evidence' };
  }

  if (!policy.automaticRoutingEnabled) {
    return { kind: 'needs_review', reason: 'automatic_routing_disabled' };
  }

  if (!policy.permittedActions.has(proposal.action)) {
    return { kind: 'needs_review', reason: 'action_not_permitted' };
  }

  if (proposal.confidence < policy.minimumConfidence) {
    return { kind: 'needs_review', reason: 'low_confidence' };
  }

  return {
    kind: 'authorized',
    action: proposal.action,
    evidence: proposal.evidence,
    policyVersion: policy.version,
  };
}

What the code guarantees

The important feature is not the syntax. It is the location of authority. The model may propose security_queue, but it cannot grant itself permission to use that route. High confidence cannot enable a feature that business rules have disabled. Missing evidence cannot be repaired by optimism.

The eight layers of the shell

The shell is a way to organize responsibility, not a software library. Its implementation will vary, but eight responsibilities recur in serious systems.

1. Check the request (input contract)

Check the request's size, type, required information, who made it and what that person is allowed to do before sending anything to the model. This is the input contract. An expensive model is not an input-checking service.

2. Record what the model saw (context assembly)

Record the instructions, business records, rule versions and documents given to the model. This is context assembly. Without that record, two apparently identical requests may produce decisions that cannot be explained or reconstructed later.

3. Limit time, cost and retries (invocation budget)

Set explicit limits for waiting time, retry attempts, generated text, simultaneous requests and cost. Engineers call this an invocation budget. “Call the model until it works” is not a failure policy.

4. Check the response format (structural validation)

Require the declared response shape. This is structural validation. Reject unknown result types, missing fields and malformed values. Do not let response-reading software silently invent values for fields that affect a decision.

5. Check whether the claims are supported (semantic validation)

Check the model's claims against material the application can verify. This is semantic validation. Confirm that referenced records exist, quoted evidence is real, numeric ranges make sense and the proposed action fits the current situation.

6. Apply business rules (policy decision)

Apply permissions, account settings, risk limits, enabled features and rules that must always hold. This policy decision happens in ordinary code. It is where a proposal becomes permitted, rejected or sent for review.

7. Save the result safely (durable state transition)

Save important changes so they survive a restart. Engineers call this a durable state transition. Make the save idempotent: repeating the same request or receiving the same event twice must not perform the consequential action twice.

8. Keep a record and monitor the process (audit and observability)

Record the proposal, evidence, rule version, final decision and reason for any failure. Measure each stage separately. This creates an audit record and makes the process observable. A service may appear healthy while document retrieval is broken, the model connection is changing or the business-rule checks reject nearly everything.

This system-level view is consistent with the AI Risk Management Framework from the U.S. National Institute of Standards and Technology (NIST). It treats governance, measurement and management as concerns throughout a system's life. Its Core calls for documented controls around outside AI components, defined human oversight, interpretation of outputs in context and systems that can fail safely beyond their knowledge limits. The framework is broader than model quality alone.

The same boundary appears in the multinational Guidelines for Secure AI System Development. They recommend limiting the actions AI components can trigger, adding independent safety mechanisms, giving each component only the access it needs and checking model outputs before use.

“Needs review” is a real result

Many applications recognize only two results: success and failure.

When a probabilistic component is added, teams often overload one of those states. Low confidence becomes rejection. A timeout becomes “not approved.” Missing evidence becomes an empty result that later parts of the system interpret as negative.

Those mappings are convenient and wrong.

A mature system normally needs at least three outcomes:

• the evidence and business rules permit an action;
• the evidence or business rules require review;
• the system could not produce a trustworthy proposal.

The second path represents uncertainty. The third represents failure. Combining them damages both product behaviour and operations. Review queues become filled with infrastructure incidents, while users receive negative outcomes caused by unavailable dependencies.

Confidence does not solve this. A confidence score can contribute to a review rule, but it is not evidence, permission or proof that the score is reliable. A system should explain why the evidence was sufficient under particular business rules, not merely report that the model sounded certain.

Preserve the identity of failure

Suppose the model call times out. The application has several honest options:

• retry a limited number of times within a fixed time and cost budget;
• place the work in a queue that survives a service restart;
• return a retryable system error;
• continue with reduced but documented capability;
• ask a human to complete the operation.

“Return the negative business result” is not one of them.

Keeping the reason for failure matters because different people and systems can respond. A user may correct a badly formed request. An operator may investigate an outside service outage. A worker may retry a temporary error. A product owner may decide whether reduced operation is acceptable.

Flattening all of these into false or score: 0 destroys that coordination.

Giving each result a clear name keeps the reason intact as it moves through the system. It also creates better measurements: business rejection rate, insufficient-evidence rate and system-failure rate are different signals. Combining them produces a number that tells nobody what to fix.

Test the surrounding system without relying on a live model

The deterministic shell creates a large surface that can be tested without a live model. Small automated checks, commonly called unit tests, can prove that the application's rules behave as intended.

Those tests can prove that:

• an action without evidence never becomes authorized;
• a disabled capability cannot be enabled by high confidence;
• a disallowed action goes to review;
• timeouts remain technical failures and state whether retrying may help;
• repeated versions of the same operation cannot produce duplicate changes;
• every saved decision records the version of the business rules used.

Integration checks can verify the request and response formats against a controlled stand-in for the model service. Engineers often call these adapter contract tests. A small, controlled set of live checks can then detect whether the real service still accepts the schema and returns compatible responses.

That division is useful because live model tests answer a different question. They can reveal a broken connection or provider drift, meaning changes in behaviour over time. They should not be the only proof that the application's essential rules always hold.

The shell has limits

A deterministic shell does not make an AI system automatically correct merely because it was designed this way.

Checks of meaning and evidence, known as semantic validation, can be incomplete. Evidence extraction can miss relevant facts. A business rule can encode the wrong decision. Human review can become a slow queue that nobody monitors. A model can be systematically wrong while still producing perfectly formed proposals.

The point is narrower: the architecture should reveal where each responsibility lives.

The model owns uncertain interpretation. Validation owns format and evidence checks. Policy owns permission to act. Storage owns the saved business record. Operations owns recovery and monitoring. Humans own the decisions the organization has chosen not to automate.

Those boundaries make failure inspectable. They make consequential actions harder to trigger accidentally. They allow individual components to improve without silently changing the meaning of the entire product.

Eight questions for an architecture review

Use these questions before allowing a model response to change a saved business record.

1. What is the model allowed to propose?
2. What evidence must accompany that proposal?
3. Which checks establish that the evidence is real and relevant?
4. Where does the application decide whether the action is permitted, known as authorization?
5. Which explicit result represents insufficient evidence?
6. Can a timeout or malformed response become a user judgment?
7. Can the same change be retried safely without happening twice?
8. Can the decision be reconstructed from its evidence, what the model saw and the version of the business rules used?

If the answer to several of these questions is “inside the prompt,” the system probably has an authority problem.

The hard part of production AI is rarely obtaining a response. It is deciding what that response is allowed to mean.