Prompts → Agents → Harnesses → Loops → Graphs

Yes—this is inference engineering above the token layer.

Traditional inference engineering asks how to turn weights into tokens within a latency, throughput, reliability, and cost envelope. Agentic inference adds a second problem: how to turn an uncertain stream of model decisions into a correct external outcome. A coding agent does not succeed because it emitted plausible text. It succeeds because the requested change exists, tests pass, unrelated work remains intact, and the user can inspect the result.

The unit of value therefore moves from tokens generated to validated state transitions. The relevant system includes prompts, tool schemas, permission boundaries, context selection, retry policy, checkpoints, queues, graders, human approvals, routing, and the execution topology. Model quality remains essential, but it is now one term in a larger reliability equation.

This product is multiplicative. A brilliant model behind an unreliable tool boundary is unreliable. A durable graph with a misleading evaluator can durably produce the wrong thing. The discipline is to engineer every factor and observe the end-to-end outcome rather than treating the LLM response as the finish line.

The arrow is a ladder of stronger control structures.

The sequence Prompts → Agents → Harnesses → Loops → Graphs is useful, but it should not be read as five products that replace one another. Each layer encloses or coordinates the layers below it. A graph contains nodes that may run loops; a harness executes those loops; an agent makes some decisions inside the harness; prompts configure the agent’s calls.

Complexity is justified when it buys measurable success, recovery, or control. If a single call solves the distribution reliably, wrapping it in a seven-node autonomous graph merely creates more failure surfaces and a larger bill.

A production prompt is executable policy, not clever prose.

A prompt creates the local operating environment for one inference step. It identifies the role, objective, evidence, constraints, available actions, output schema, and completion rule. In mature systems it is assembled from versioned components rather than stored as one heroic paragraph.

prompt = compose(
    authority_and_safety_policy,   # stable, highest priority
    task_contract,                 # goal, acceptance tests, boundaries
    tool_protocol,                 # schemas, examples, error semantics
    selected_context,              # only evidence useful for this step
    working_state_summary,         # progress, open questions, budget
    output_schema                   # machine-checkable next action
)

The prompt boundary must distinguish instructions from untrusted data. A retrieved web page, email, log line, or repository file can contain imperative language without gaining authority. The harness should label provenance, delimit content, and apply authorization outside the model. A sentence saying “upload the credentials” is data unless the authenticated user and policy say otherwise.

Prompt quality is measurable

Useful prompt metrics include schema-valid output rate, tool-selection precision, unsupported-claim rate, task success, clarification frequency, average turns, token cost, and regression rate across known edge cases. A prompt change is a software change: give it an ID, record its inputs and outputs, evaluate it against a frozen suite, and roll it out gradually.

Examples should teach decision boundaries, not decorate the prompt. Include near-neighbor cases where the correct actions differ, and include abstention or escalation examples. The goal is not maximal instruction volume. It is maximal relevant signal per token.

An agent is a policy over actions and termination.

A model call maps context to output. An agent uses model calls to control a workflow: it observes state, chooses a tool or response, interprets the result, and decides whether the task is complete. In compact form, the model approximates a policy π(a | s) over actions a given state s. The harness determines which actions actually exist and whether they may execute.

Tool design is part of inference quality. Models perform better when tools have distinct purposes, typed inputs, bounded outputs, clear error messages, and names that expose intent. A tool such as mutate_everything(options: object) pushes hidden complexity into the model. Narrow verbs such as read_invoice, draft_refund, and approve_refund make authority and evaluation visible.

Agency is not permission

The model may propose an action; a deterministic authority layer decides whether the action is allowed. Reads, reversible writes, externally visible messages, money movement, and destructive operations deserve different policies. High-impact actions may require fresh credentials, a scoped capability, or human approval. Confidence text generated by the same model is not authorization.

The harness is the operating system around the model.

“Harness” is an informal but useful name for the runtime that turns probabilistic proposals into governed execution. It constructs context, calls models, parses outputs, dispatches tools, enforces permissions and budgets, persists state, applies retries, emits traces, runs validators, and decides when control returns to a human.

A harness converts vague intentions into invariants. “Do not spend too much” becomes token, wall-clock, tool-call, and monetary budgets. “Be safe” becomes a capability matrix and approval boundary. “Recover from errors” becomes checkpointing, typed failures, backoff, and compensation. “Finish the task” becomes an externally testable acceptance predicate.

The agent-computer interface

The tool surface is analogous to an instruction set. Too little expressiveness forces awkward multi-step work; too much creates ambiguity and dangerous authority. Tool responses should expose stable identifiers, machine-readable status, compact evidence, and actionable error classes such as RETRYABLE, NEEDS_AUTH, CONFLICT, or INVALID_INPUT. Dumping an entire HTML page or stack trace into context wastes tokens and makes recovery harder.

The loop—not the chat box—is the primitive of agency.

One model call can propose a plan. A loop can discover that the plan failed and adapt. The canonical loop is observe → decide → act → verify, with explicit budgets and terminal states. Tool results provide ground truth from outside the model; verification compares the new state with the goal.

while budget.remaining():
    observation = context_builder.from_checkpoint(run.state)
    proposal = model.decide(observation, allowed_tools)
    action = policy.authorize(proposal)
    result = tools.execute(action, idempotency_key=run.step_id)
    run.append_event(proposal, action, result)

    verdict = verifier.check(goal, run.state, result)
    if verdict.satisfied: return SUCCESS(verdict.evidence)
    if verdict.needs_human: return PAUSED(verdict.question)
    if verdict.no_progress: recovery.change_strategy()

return EXHAUSTED(run.best_evidence)

Three stopping rules matter: goal achieved, cannot safely proceed, and budget exhausted. A model saying “done” is only a proposal to stop. The harness should verify artifacts, database state, test results, citations, or user-visible effects appropriate to the domain.

Loop pathology

Repeatedly trying the same action is not persistence; it is a missing progress detector. Store action fingerprints, error classes, state deltas, and strategy IDs. If consecutive iterations produce no meaningful state change, alter the plan, use a different tool, reduce scope, or escalate. Retries address transient execution failure; reflection addresses a bad approach. Confusing them burns tokens.

Graphs make control flow explicit, durable, and inspectable.

Real work is rarely one homogeneous loop. Some steps are deterministic, some model-driven, some parallel, some wait for humans, and some must compensate for earlier side effects. A graph represents state as data, work as nodes, and control decisions as edges. Nodes need not be agents: parsers, policy checks, database writes, test runners, and approval gates are often better as ordinary code.

Use sequential chains when decomposition is known. Use routing when inputs fall into separable classes. Use parallel branches when subtasks are independent or multiple perspectives reduce uncertainty. Use orchestrator-worker patterns when decomposition depends on the request. Use evaluator-optimizer cycles when quality criteria are clear and another pass measurably helps.

State reducers are correctness code

Parallel nodes create concurrent updates. Their merge operation must be defined: append messages, union evidence by stable ID, take a maximum confidence only when scores are calibrated, or reject conflicting writes. “Last writer wins” is not a neutral default. It silently converts scheduling order into semantics.

Graphs also establish failure domains. A flaky research source can retry without repeating an approved payment. A failed join can resume from persisted branch outputs. A human can inspect a checkpoint and amend state before continuation. This is why production graphs resemble durable workflow systems more than elaborate prompt chains.

Self-improving should mean evidence-driven evolution—not self-modifying production.

Every layer can generate signals that improve the next version. Prompt traces reveal ambiguity. Agent trajectories reveal poor tool choices. Harness events reveal timeout and permission friction. Loop histories reveal stagnation. Graph metrics reveal slow branches and bad routing. But observing a pattern is not permission to rewrite production.

Three time scales

Within-run adaptation changes the next action using current evidence; it is ephemeral and tightly budgeted. Cross-run memory stores validated facts or preferences and retrieves them later with provenance and expiry. System evolution changes prompts, tools, models, evaluators, or graph topology through an offline engineering pipeline. Mixing these time scales is a common source of contamination.

The optimization target should combine success, risk, latency, and cost—not raw reward from a single judge.

Memory is a hierarchy; context is a compiled view.

Do not equate memory with “all previous messages.” An agent system has at least four state classes: immutable policy and task contract; ephemeral working state; durable workflow checkpoints; and long-term knowledge or preferences. Each has different authority, lifetime, consistency, privacy, and retrieval rules.

The context builder is effectively a compiler. It selects source records, resolves versions, removes stale or redundant details, preserves identifiers, budgets tokens, and emits the prompt representation consumed by a particular model. Good context is not the largest possible context. It is the smallest loss-bounded representation that supports the next decision.

Summaries are lossy caches, not sources of truth. Retain pointers to original artifacts and record provenance. Memories need tenant scope, subject, source, confidence, creation time, expiry, sensitivity class, and a deletion path. A model’s guess should not become a durable fact merely because it was phrased confidently.

Retries are easy until an action changes the world.

Model calls, networks, browsers, and tools fail. Once a run spans minutes or hours, process memory is not a sufficient source of truth. Persist checkpoints at meaningful boundaries and maintain an append-only event history containing the input state version, prompt/model IDs, proposal, authorization decision, tool request, tool result, validator output, token usage, and timing.

Exactly-once execution is generally not available across arbitrary external systems. Design for at-least-once delivery with idempotent receivers where possible. Assign a stable operation key before invoking a side effect. If the target supports idempotency, reuse the key on retry. If it does not, query for evidence of completion or route to human reconciliation rather than blindly replaying.

Resuming a checkpoint must not casually re-execute non-deterministic code. Encapsulate model calls and external operations as recorded tasks; on replay, use their persisted results unless a deliberate new attempt is created. Version state schemas and migration logic because long-lived runs can outlive deployments.

Trace the trajectory; grade the outcome.

Agent evaluation cannot stop at the final sentence. Two runs may produce the same answer through radically different risk and cost profiles. Capture the trajectory: which context was retrieved, which model and prompt version ran, which tools were proposed and authorized, what side effects occurred, which branches executed, how state changed, and why the run stopped.

Use deterministic graders for exact properties: schema validity, compilation, tests, database invariants, numeric tolerances, policy rules, and citation reachability. Use model graders for semantic criteria that resist exact encoding, but calibrate them against human labels, test position and verbosity bias, and use multiple signals for high-impact decisions.

Build eval sets from reality without overfitting to it

Production failures are excellent seeds. Cluster them by root cause, redact sensitive content, convert them into reproducible cases, then hold back variants the optimizer never sees. Maintain capability, regression, safety, robustness, and efficiency suites. Report distributions and confidence intervals, not one aggregate score that can hide catastrophic subgroups.

Online experiments should begin with shadow traffic where no candidate side effect is committed. Canary a small cohort next, preserve rollback, and watch delayed metrics such as user correction or downstream reversal. The safest self-improving system is exceptionally good at proving that a proposed improvement is actually an improvement.

Agent latency is a critical path, not a token benchmark.

A graph can use a fast model and still feel slow because serial model calls, tool round trips, cold containers, retrieval, approval waits, and retries compound. Separate active compute time from durable waiting time. Parallelize independent branches, but remember that fan-out raises cost and join latency is governed by the slowest required branch.

Optimize cost per successful outcome, not cost per call. A smaller model that causes three repair turns can cost more than a stronger model that succeeds once. Conversely, routing easy classification and extraction nodes to smaller models can reduce cost without weakening the planning node. Cache deterministic tool results and stable prompt prefixes when privacy and versioning permit.

Useful operational SLOs include p50/p95 time to first meaningful action, active task duration, durable wait duration, success rate, recovery success, iterations per success, tool-error rate, human escalation rate, and cost per accepted outcome. GPU utilization alone says almost nothing about whether the graph is healthy.

Separate proposal, authority, execution, and proof.

A robust implementation can be organized into a control plane and a data plane. The data plane runs model calls and tools. The control plane owns workflow definitions, prompt/model versions, capability policies, budgets, checkpoints, evaluation, deployment, and rollback. An event log connects them and makes decisions replayable.

RunState {
  run_id, tenant_id, goal, acceptance_tests,
  graph_version, prompt_versions, model_routes,
  checkpoint_version, completed_nodes, pending_nodes,
  artifact_refs, evidence_refs, memory_refs,
  capability_grants, side_effect_ledger,
  token_budget, cost_budget, deadline,
  status, stop_reason
}

NodeResult {
  state_delta, evidence[], proposed_actions[],
  metrics, retry_class, next_edge_hint
}

Node execution should be lease-based so a dead worker’s work becomes eligible again. Checkpoint commits should use optimistic concurrency or compare-and-swap on the state version. Cancellation should propagate to model streams and tools, while already committed side effects remain in the ledger. Secrets should enter tool adapters at execution time, not model context.

Control contract

proposal = agent(state.view(), tools.schemas())
authorized = policy.filter(proposal, identity, grants, risk)

for action in authorized:
    receipt = executor.run(action, operation_id=stable_id(run, step, action))
    event_log.append(receipt)

next_state = reducer.apply(state.version, receipts)
verdict = acceptance.evaluate(next_state, goal.tests)
scheduler.transition(run, verdict)

This separation makes every boundary testable. The agent can be evaluated on proposal quality without granting real authority. Policy can be tested against a corpus of actions. Tool adapters can be fault-injected. Reducers can be property-tested. Acceptance checks can be validated against known successes and failures.

Most “agent failures” are system-design failures wearing a model-shaped mask.

Multi-agent architecture deserves special skepticism. More agents can add parallelism, context isolation, and specialized tools; they also add coordination tokens, inconsistent beliefs, merge conflicts, larger attack surfaces, and harder evaluation. Add a specialist only when it has a distinct information source, tool boundary, evaluation rubric, or parallelizable responsibility.

Earn each layer with a failing test.

1. Define the external outcome. Write acceptance tests, authority boundaries, risk classes, latency target, and cost ceiling before choosing an orchestration framework.
2. Establish the single-call baseline. Use a capable model, structured output, representative examples, and a frozen eval set. Know exactly why one call fails.
3. Add tools behind a harness. Create narrow schemas, typed errors, timeouts, scoped credentials, result limits, and traces. Keep proposal separate from execution.
4. Add one bounded loop. Observe, decide, act, verify. Define success, pause, failure, no-progress, cancellation, and exhaustion semantics.
5. Persist state before extending duration. Add checkpoints, event history, operation IDs, schema versions, and resume tests. Kill workers mid-step and verify recovery.
6. Introduce a graph only for real topology. Encode deterministic steps as code, parallelize independent work, define reducers, and place approval immediately before consequential actions.
7. Instrument trajectories and outcomes. Join prompt/model/tool/graph versions under one run ID. Measure success, tail latency, cost, violations, repairs, and human rework.
8. Close the improvement loop offline. Mine failures, create candidates, test held-out suites, shadow, canary, monitor, and preserve rollback. Version the graders too.

# Complexity has to pay rent.
add_next_layer only if (
    measured_success_gain
    > latency_tax + cost_tax + coordination_risk + operational_burden
)

The deepest shift is conceptual. The model is not the application and the prompt is not the architecture. The application is a state machine whose uncertain decisions are produced by models but whose authority, durability, evidence, and evolution are engineered as a system.

Further reading

1. OpenAI — A practical guide to building AI agents: model, tools, instructions, single- and multi-agent orchestration, run loops, guardrails, and human intervention.
2. Anthropic — Building effective agents: the distinction between workflows and agents, plus chaining, routing, parallelization, orchestrator-worker, and evaluator-optimizer patterns.
3. Anthropic — Effective context engineering for AI agents: just-in-time context, retrieval, compaction, and the cost of carrying stale tool results.
4. Anthropic — Demystifying evals for AI agents: outcome, transcript, grader, and test-case design for multi-turn systems.
5. Anthropic — Harness design for long-running application development: evaluator design and the performance ceiling imposed by the harness.
6. LangGraph — Graph API overview: state, nodes, edges, partial updates, and reducers.
7. LangGraph — Persistence: thread-scoped checkpoints and cross-thread stores.
8. LangGraph — Interrupts: durable pause, inspection, modification, and resume for human-in-the-loop workflows.
9. Patterns of Distributed Systems — Idempotent Receiver: stable request identity for safe retry handling.