A harness is the code sitting between the user's request and the model's tool calls: the loop that assembles context, dispatches actions, catches failures, and decides when to stop. Concretely, this means a version-controlled spec file (AGENTS.md, CLAUDE.md) that states task requirements as a contract the agent re-reads every run, and a Plan-Act-Observe loop where the model proposes a plan, something gates it (a human, a scripted check, or a second model), and only then does it act and observe the result before looping again.

The gate is doing the real work. Deterministic verification — lint, unit tests, schema validation against a real spec — sits between a proposed diff and a merge, so the agent's own confidence never becomes the pass condition. Multi-agent decomposition (planner, executor, critic as separate contexts) exists for the same reason: it keeps any single context window from accumulating the stale, contradictory state that causes context rot on long-running tasks, rather than to parallelize work.

Fail-safe boundaries close the loop. MCP-style scoping — a personal token that inherits the calling user's own permissions instead of a blanket service key — bounds the blast radius of a bad tool call. A hard iteration cap and a dollar budget, enforced by the harness and not by the model noticing it should stop, bound the blast radius of a bad loop.

request ──► spec (AGENTS.md / CLAUDE.md, re-read every run)
                       │
                       ▼
      ┌────────────────────────────┐
   ┌─►│       PLAN  (model)        │
   │  └──────────────┬─────────────┘
   │                 ▼
   │  ┌────────────────────────────┐
   │  │ GATE — lint / tests /      │
   │  │ schema, or a human /       │
   │  │ second-model critic        │
   │  └───────┬──────────┬─────────┘
   │      fail│          │pass
   └──────────┘          ▼
              ┌────────────────────────────┐
              │       ACT  (tool call)     │
              └──────────────┬─────────────┘
                              ▼
              ┌────────────────────────────┐
              │          OBSERVE           │
              └──────────────┬─────────────┘
                              │
                    loop back to PLAN
   ────────────────────────────────────────────
   hard iteration cap + $ budget: enforced by the
   harness, checked before each tool call — not
   after, and not by the model noticing it's stuck
    

The numbers

Where the harness ceiling ends

The evidence doesn't say scaffolding is unlimited: once a competent harness exists, swapping the underlying model is reported to dominate every other lever — the harness buys roughly one model generation of headroom (a 2–6x multiplier over a naive baseline), then hits a model-bound wall on multi-turn coherence and long-horizon autonomy.

A prompt is one message. Context is everything the model sees at generation time: system instructions, retrieved documents, tool schemas, tool outputs, and conversation history, all competing for the same fixed token budget. Prompt engineering optimizes wording within that budget; context engineering decides what's even in the budget. Put stable, high-priority instructions at the top and the immediate task at the bottom, with retrieved data and tool results in between behind explicit delimiters (XML tags or markdown headers) so the model can tell a system rule from a quoted document.

The reason this matters more than phrasing now is the lost-in-the-middle effect: models attend well to the start and end of a long context and poorly to the center, so a critical constraint buried mid-window gets silently dropped even though it's technically "in context." This produces three distinct failure modes worth naming separately because they need different fixes. Context distraction is irrelevant history or bloated tool JSON diluting attention on the actual task. Context rot is the positional loss described above — the fact was there, but in the wrong place. Context clash is stale memory or a contradictory retrieved chunk overriding current state, e.g., an old ticket status resurfacing after the ticket was closed.

The fix isn't a bigger window. A harness — the code around the model, not the text file — should reserve headroom (roughly 10% of the window unused), trigger compaction once usage crosses ~60%, and summarize old turns into a state document instead of replaying raw transcripts. On the retrieval side, a reranker (cross-encoder) narrows a broad recall set down to the few chunks actually worth the tokens, and tool outputs get field-extracted rather than dumped whole.

CONTEXT WINDOW — fixed token budget, filled top to bottom
┌──────────────────────────────────────────────────────────────┐
│ SYSTEM INSTRUCTIONS (stable, top)            attention: HIGH │
├──────────────────────────────────────────────────────────────┤
│ <docs> retrieved chunks (reranked) </docs>                   │
│ <tools> tool schemas + outputs </tools>       attention: LOW │
│   ↳ distraction / rot / clash live in this band              │
├──────────────────────────────────────────────────────────────┤
│ IMMEDIATE TASK (bottom, read last)           attention: HIGH │
╞══════════════════════════════════════════════════════════════╡
│ headroom — reserved empty, never filled (~10% of budget)     │
└──────────────────────────────────────────────────────────────┘
  0%              60% → compaction trigger          90%  100%

The numbers

Measuring it

There's no single agreed metric for "context quality." Practitioners instead A/B task success rate across configurations — raw chat, system/project brief only, retrieval added, retrieval plus summarized history — and log context diffs between agent steps to see exactly what changed. The cheapest diagnostic: check whether retrieved chunks are actually cited in the output. If a third or more never show up in the final answer, you're paying retrieval latency and tokens for nothing.

Prompt caching lives at the inference layer. On the first request, the provider stores the KV tensors for a stable prefix (system prompt, tool schemas, few-shot examples). Any later request whose prefix matches that stored one token-for-token skips reprocessing it, cutting both cost and time-to-first-token. The match is exact: change a single character before the cached boundary — a timestamp, a session ID, a reordered tool list — and the whole prefix misses. That's why the standard pattern is "static content first, dynamic content last": put the system prompt, docs, and tool defs up top under cache_control, and put the user's message and any per-request variables after the cache boundary.

Semantic caching lives at the application layer, ahead of the model entirely. Each incoming query gets embedded, and a vector lookup checks whether a sufficiently similar query has been answered before. Above a cosine-similarity threshold — typically 0.92-0.95 — the stored response is returned and the model is never called. This is where the two techniques stop being comparable: prompt caching only ever changes cost and latency, never correctness, because the model still runs and still sees the full prompt. Semantic caching can change the answer, because it substitutes a different (past) response for a new (current) query based on embedding proximity, not identity.

Because they sit at different layers, they stack rather than compete: exact-match check first (free, zero risk), then semantic match (cheap, some risk), then a real model call with cache_control enabled on the prefix as the fallback. Skipping straight to semantic caching without prompt caching underneath leaves money on the table on every cache miss.

  incoming query
        │
        ▼
  ┌────────────────────────┐  hit → return cached text,
  │ 1. exact match cache   │──$0, ~0ms, zero model calls
  └────────────────────────┘
        │ miss
        ▼
  ┌────────────────────────┐  hit (cos ≥ 0.92–0.95) →
  │ 2. semantic cache      │  return a PAST answer —
  └────────────────────────┘  risk: wrong intent matched
        │ miss
        ▼
  ┌───────────────────────────────────────────────────────────────┐
  │ 3. real model call                                            │
  │ ┌──────────────────────────────┬──────────────────────────┐   │
  │ │ cached prefix [cache_ctl]    │ dynamic suffix           │   │
  │ │ system+tools+few-shot        │ user msg, session id,    │   │
  │ │ ≥1024 tok, 90% off on hit    │ timestamp — one moved    │   │
  │ │ (Sonnet/Opus min size)       │ here = 0% hit rate       │   │
  │ └──────────────────────────────┴──────────────────────────┘   │
  └───────────────────────────────────────────────────────────────┘

The numbers

Setup cost

Prompt caching is close to free to add: mark the stable prefix with cache_control: {type: "ephemeral"} and reorder the prompt. Semantic caching requires an embedding pipeline, a vector store (Redis, Pinecone, etc.), threshold tuning per use case, and — for multi-tenant systems — cache keys namespaced by tenant/user ID so one customer's cached answer can't leak into another's session.

During autoregressive decoding, generating token N needs the keys and values of every prior token; recomputing them each step is quadratic, so engines cache them per layer, per head. At long context this cache consumes 70-90% of GPU memory and dominates wall-clock time. Before vLLM's PagedAttention, serving systems reserved one contiguous block per request sized for the worst-case output length — a request that generates 200 tokens against an 8K max context locks up 97% of its reservation, unusable by any other request. The vLLM paper measured only 20-38% effective utilization under that scheme.

PagedAttention borrows virtual-memory paging: physical KV storage is split into fixed-size blocks (commonly 16 tokens), and each sequence holds a logical block table mapping onto them. Blocks fill on demand, get freed as sequences finish, and there's no requirement that a sequence's blocks sit contiguously — fragmentation drops and utilization climbs above 96% (waste under 4%). This is table-stakes now (on by default in vLLM), so the differentiator at the deployment level is sizing the cache pool and picking the right eviction and reuse policy, not enabling the feature.

Reuse compounds the win. Prefix caching (vLLM's automatic prefix caching, SGLang's RadixAttention) stores KV blocks for repeated prefixes — system prompts, RAG context, multi-turn history — keyed by content hash so a new request with a matching prefix skips prefill entirely for that span. Prefill is the expensive phase (full attention over the whole prompt), so a cache hit on a long shared prefix can turn a multi-second prefill into a near-zero one. Production agentic traces (Mooncake's Qwen-Bailian analysis) show hit rates above 95% when caches are retained long enough, since tool-using agents append small deltas to long, reused contexts across tens or hundreds of rounds.

BEFORE PagedAttention: one contiguous block, sized for worst case
+----------------------------------------------------------------+
| used (200 tok) |        reserved-but-idle (rest of 8K max)     |
+----------------------------------------------------------------+
  3% used -> 97% locked, unusable by anyone else (util: 20-38%)

AFTER: fixed-size blocks (16 tok) + per-sequence block table
  Req A table                Req B table (shares A's prefix)
   idx -> phys                idx -> phys
    0  ->  7  -+                0  ->  7  -+  same phys block:
    1  ->  2  -+ shared prefix  1  ->  2  -+  prefix-cache hit,
    2  ->  9     private tail   2  ->  4      no prefill needed
                                3  ->  1      private tail

  Physical pool: [7][2][9][4][1][5][8][3][6][0] ...   ~90% util
  Sequence finishes/evicts -> its blocks return to the free pool

The numbers

Eviction and offload policy

Default to LRU eviction. If monitoring shows high-value prefixes (long system prompts, popular RAG chunks) getting evicted before reuse, move to frequency- or priority-aware eviction instead. When GPU memory is exhausted, tiered offload to CPU DRAM or NVMe (e.g. Mooncake's SSD-offload path) extends effective capacity — but only offload when transfer-plus-restore latency beats the cost of recomputing that span from scratch. Offloading is a capacity lever, not a speed lever: it lets you hold more sessions, it doesn't make any single one faster.

Every request has two phases with opposite arithmetic. Prefill runs the full prompt through the model in a single forward pass, reusing each loaded weight across every token in the batch — that reuse pushes the GPU into its compute ceiling (TFLOP/s-bound). Decode generates one token at a time: each step reads the entire weight set and the growing KV cache but does almost no math per byte moved, so it sits on the memory-bandwidth ceiling (TB/s-bound) instead. This is the "roofline" split — same chip, two completely different limiters — and it's why a change that speeds up one phase (bigger batches, more parallel math) routinely does nothing or actively hurts the other.

The practical failure is contention: a scheduler that lets a long prefill run on the same GPU as active decode streams stalls every in-flight generation until the prefill finishes, producing visible stutter in inter-token latency (ITL). vLLM, SGLang, and NVIDIA Dynamo handle this with continuous batching plus chunked prefill: a long prompt gets sliced into chunks (governed by max_num_batched_tokens) and interleaved step-by-step with decode work, so decode never waits behind a full prefill pass — this is "stall-free batching."

At high scale, teams go further and physically separate the phases: prefill runs on one GPU pool, decode on another, and the KV cache produced by prefill is shipped over InfiniBand/RDMA to the decode node before generation starts. This disaggregated serving eliminates GPU-level contention entirely, since prefill can never block a decode step it doesn't share hardware with — but it introduces a new bottleneck: if the KV-cache transfer is slow relative to prefill compute, the network hop eats the latency win you disaggregated for.

COMPUTE-BOUND (TFLOP/s)              MEMORY-BOUND (TB/s)
┌───────────────┐                    one token, full
│    PREFILL    │                    weights + KV reread
│  whole prompt │                    ┌───┐┌───┐┌───┐
│   one pass    │                    │dec││dec││dec│
└───────────────┘                    └───┘└───┘└───┘

Shared GPU, no chunking → contention:
┌─────────────────────────────┐
│      prefill runs (long)     │   decode STALLS
└─────────────────────────────┘   → ITL stutter
                                ┌──┐┌──┐┌──┐
                                │d ││d ││d │
                                └──┘└──┘└──┘

Chunked prefill (max_num_batched_tokens slices it):
┌──┐┌──┐┌──┐┌──┐┌──┐┌──┐┌──┐┌──┐
│pf││d ││pf││d ││pf││d ││pf││d │  decode never waits
└──┘└──┘└──┘└──┘└──┘└──┘└──┘└──┘  behind a full pass

Disaggregated serving:
┌─────────────┐   KV cache    ┌─────────────┐
│ prefill GPU │ ── RDMA ────→ │ decode GPU  │
│   pool A    │ (new bottle-  │   pool B    │
└─────────────┘  neck here)   └─────────────┘
    

The numbers

When to disaggregate

Chunked prefill on shared hardware is the right default for most workloads — it's a scheduling change, not an infrastructure change. Disaggregation only pays off once request volume is high enough that prefill-decode contention is the dominant tail-latency cost, and you have RDMA-class networking to move the KV cache without re-introducing the stall you removed.

Static batching runs a fixed group of requests together and refuses to release any of them until the slowest sequence finishes. Mix a 15-token chat reply with an 800-token code generation in the same batch and the GPU spends 785 iterations recomputing forward passes for a request that already has its answer, just to keep the batch shape uniform. Continuous batching (iteration-level scheduling, in-flight batching) removes this constraint: after every forward pass, finished sequences exit and queued sequences enter the batch, so a GPU slot is idle for at most one decode step instead of an entire batch's lifetime.

The scheduler needs somewhere to put each new sequence's KV cache without stalling to defragment memory, which is what PagedAttention solves. Instead of pre-allocating one contiguous tensor sized to a request's maximum possible length, the KV cache for each sequence is split into fixed-size blocks (16 tokens each in vLLM's default) and a per-request block table maps logical block indices to physical blocks pulled from a shared pool on demand — the same trick an OS uses to map virtual pages to physical frames. Before this, profiling showed only 20.4-38.2% of reserved KV cache memory held real token state; the rest was internal fragmentation, external fragmentation, and over-reservation for sequences that never reached their max length.

Prefix caching stacks on top of both: if requests share a prompt prefix — a system prompt, a few-shot block, a long document reused across turns — the blocks for that prefix are hashed and reused across requests instead of recomputed. On workloads with shared prefixes this alone adds another 30-60% throughput on top of continuous batching and paging.

CONTINUOUS BATCHING — GPU batch slots across decode iterations

          t0     t1     t2     t3     t4
 slot 0 [ A  ] [ A  ] [ A✓ ] [ C  ] [ C  ]   A finishes at t2
 slot 1 [ B  ] [ B  ] [ B  ] [ B  ] [ B  ]   B keeps decoding
 slot 2 [ C  ] [ C  ] [ C  ] [ C  ] [ C  ]   C admitted at t0

 static batching: whole batch blocked until slowest (B) exits —
 A's slot sits idle from t2 to end even though A is long done.
 continuous batching: A's slot is handed to queued work at t3.

PAGEDATTENTION — per-request block table → shared physical pool

 request A logical blocks      physical KV pages (16 tok/page)
   0 ───────────────────────►  page 7   (A, tok 0-15)
   1 ───────────────────────►  page 2   (A, tok 16-31)
   2 ───────────────────────►  page 9   (A, tok 32-47)
                                page 4   (B, tok 0-15)
                                page 0   (free — next request)
    

The numbers

For why prefill and decode have opposite bottlenecks and how chunked prefill and disaggregation address that, see the prefill vs. decode latency section.

Autoregressive decode is memory-bandwidth bound: on an H100 with 3.35 TB/s HBM, a 70B model in FP16 caps out near 24 tokens/sec/request purely from reloading weights every step. Quantization (e.g. AWQ W4A16) attacks this by shrinking the bytes moved per step — dropping one precision level typically buys 30-50% better throughput, and INT8 KV cache shrinks the other half of the bandwidth bill. Speculative decoding attacks the same wall from the schedule side: a small draft model proposes K tokens, the target model verifies all K in one forward pass, and accepted tokens come out for the cost of a single target step instead of K sequential ones.

These compose multiplicatively, not additively, per the ML-SpecQD study (arXiv 2503.13565): quantizing the target model frees GPU memory, which raises the batch size ceiling, which means more draft tokens get verified per HBM read. But that compounding window is narrow: speculative decoding's own gain is 2-3x at batch size 1 and inverts to a net loss (~0.92x, reported in a real voice-ordering deployment) by batch size 12 — well below most production concurrency. The catch is that speculative decoding's win depends on staying memory-bound. Push batch size high enough and the target model's verification pass itself becomes compute-bound — at that point extra draft tokens just add wasted FLOPs, and speculation can net-negative your throughput. Treat it as a low-batch, latency-sensitive lever (voice, single-user chat), not a blanket throughput multiplier — the memory-bound regime where quantization's compounding helps closes fast as concurrency rises.

Draft model sizing matters more than draft model cleverness: a draft that's 1/10 to 1/50 the size of the target keeps its own forward pass cheap enough to be "free" relative to the savings from higher acceptance. Newer draft heads like EAGLE-3 fuse the draft step into the target model's own hidden states (no separate small model to serve), and AWS's P-EAGLE generates all K draft tokens in a single forward pass rather than K sequential ones, stacking another 1.1-1.4x on top.

  QUANTIZATION → SPECULATIVE DECODING: HOW THE WINS COMPOUND

    4-bit quant (AWQ) on target model
          │
          ↓  frees GPU memory (fewer bytes/param per step)
    higher max batch size fits in VRAM
          │
          ↓  more sequences share each HBM weight read
    more draft tokens verified per read      ← MULTIPLICATIVE
          │
          ↓  holds only while still memory-bandwidth-bound
    ── crossover: verify pass saturates GPU FLOPs (compute-bound) ──
          │
          ↓
    extra draft tokens = wasted FLOPs → speedup can go NEGATIVE
    

The numbers

Where distillation fits

Distillation is orthogonal to both: it trains a permanently smaller model to mimic the target's behavior, trading it in wholesale rather than accelerating the original. Use it when you can tolerate a fixed quality ceiling set at training time; use quantization + speculative decoding when you need the target model's exact output distribution preserved (speculative decoding's verify step guarantees this) but served faster.

Inference is memory-bandwidth bound, not compute bound: the GPU spends most of a decode step waiting on weights to stream off HBM, so halving bytes-per-weight roughly halves that wait. FP8 (e4m3/e5m2) runs natively on H100/H200/B200 tensor cores, needs no calibration dance, and sits within ~0.5% perplexity of BF16 — it's the 2026 default whenever that hardware is available. INT4 is a harder trade: 4x memory savings over FP16, but you're now choosing a quantization method, not just a bit-width.

GPTQ quantizes layer-by-layer using a Hessian-based least-squares solve per layer, minimizing local reconstruction error against a calibration set. AWQ instead observes activation magnitudes and protects the top ~1% of "salient" weight channels by scaling them before rounding, leaving the rest to quantize coarsely. That protection is why AWQ tends to beat GPTQ on code, math, and reasoning at 4-bit — but GPTQ still wins when you need sub-4-bit (3-bit, 2-bit) compression, since AWQ's salient-channel trick degrades faster below 4 bits.

The format is not the throughput number — the kernel is. AWQ weights run through a naive dequant-then-matmul kernel at a fraction of the speed of AWQ run through Marlin, a kernel purpose-built for INT4xFP16 mixed-precision GEMM. Two deployments both labeled "AWQ INT4" can differ by 2x in tok/s purely on kernel choice — always check what your serving stack (vLLM, TensorRT-LLM, SGLang) actually dispatches to.

DECODE STEP -- memory-bandwidth bound, not compute bound
  HBM --weights--> [ wait ] --> compute --> next token
                     ^ bytes/weight sets the wait

BIT-WIDTH CHOICE
  FP8 (H100/H200/B200)         INT4 -- pick a METHOD:
  native tensor cores                    |
  no calibration              +----------+----------+
  ~0.5% ppl hit              GPTQ                   AWQ
                        Hessian solve,       protect top ~1% salient
                        per-layer min        channels (by activation
                        recon error          magnitude), round rest
                              |                       |
                        wins sub-4-bit        wins at 4-bit
                        (3-bit, 2-bit)         (code/math/reasoning)
                              +----------+----------+
                                         v
                 KERNEL DISPATCH -- the real throughput lever
       naive dequant+matmul  <-- 2x slower -->  Marlin INT4xFP16 GEMM

The numbers

Where it breaks

Calibration set mismatch is the silent killer: GPTQ and AWQ both fit their scales to a calibration sample, and if you calibrate on general chat data but deploy on code or math, the salient weights the method chose to protect are the wrong ones. Long-context workloads (32k+ tokens) are the other blind spot — INT4's numerical noise accumulates over a long forward pass in ways a short-context MMLU eval never surfaces, so a model that looks fine at 4k tokens can visibly degrade at 32k.

Three techniques generate structured output, and they are not interchangeable. Prompted JSON ("return valid JSON") relies on the model following instructions and parses at roughly 85-95%. Function/tool calling ties output to a schema the provider trained the model to fill, pushing parse rates to 98-99%. Constrained decoding restricts the sampler to only tokens that satisfy a grammar — strict: true in OpenAI's API, Anthropic's strict tool use — and gets you to effectively 100% syntactic conformance because invalid tokens are masked out of the logits before sampling, not caught after the fact.

That 100% number is where teams get complacent. Constrained decoding guarantees the shape is right, not that the values are right. A model forced to emit a valid enum will pick a plausible-looking member of that enum rather than refuse or hedge, because refusal isn't a legal token under the grammar. The failure mode shifts from a parser exception to a confident hallucination sitting quietly in a column that passed every schema check — a tax rate that doesn't exist, a total that doesn't sum from its line items, a required field back-filled with a default no one asked for.

The fix is a second, independent validation layer sitting after the schema gate: a Pydantic or Zod pass that checks business logic the JSON Schema can't express — cross-field consistency, arithmetic that has to reconcile, ranges tighter than the type system allows. When that layer rejects a payload, feed the specific validator error message back to the model as the correction instruction, not a generic "try again." Bound this to one retry. If the second attempt still fails, the problem is the prompt or the schema, not bad luck, and a third attempt just burns tokens finding that out again.

model output
      │
      ▼
┌─────────────────┐   constrained decoding / strict tool-use
│  schema gate     │   ~100% syntactic conformance
└────────┬─────────┘   (shape guaranteed — VALUES unchecked)
         │
         ▼
┌───────────────────────┐  Pydantic/Zod: cross-field math,
│ business-logic check  │  ranges, required-field defaults
└───────────┬───────────┘
            │
   pass ────┼──── fail (attempt 1)
     │              │
     ▼              ▼
   ship      validator error → fed back to model → retry ×1
                    │
           pass ────┼──── fail (attempt 2)
             │              │
             ▼              ▼
           ship     STOP — prompt/schema bug, not bad luck
                    (escalate; a 3rd call just burns tokens)
    

The numbers

Termination signals, not just parse success

Before you hand a response to the validator, check finish_reason (OpenAI-style) or stop_reason (Anthropic-style). Three outcomes need three different branches, not one retry path: a length/max_tokens stop means the output budget was too small — raise it or split the task, don't just resubmit the same request. A refusal stop means the model made a policy decision — retrying with the same prompt wastes a call; route to a fallback model or a human queue instead. And a clean stop with a field silently empty is not a "use the default" case — log it as a data point, because a rising rate of empty fields is usually the first symptom of prompt or schema drift, well before your conformance dashboard notices anything.

When you instrument a real agent, the split is stark: the model usually picks the right tool and a defensible set of arguments. What breaks is everything downstream — a field typed as an integer arrives as a string, an optional field the schema allows is missing 12% of the time and the backend 500s, or a transient timeout gets retried by both the agent framework's loop and the underlying HTTP client, so a process_refund call that should run once runs three times. None of this is a reasoning failure; it's a contract failure at the boundary between a non-deterministic planner and deterministic backend code.

The fix is schema-first design enforced at the boundary, not in the prompt. Use JSON Schema with additionalProperties: false and enum for every closed-set field, validate with Pydantic or AJV before the tool body ever executes, and on failure return a structured error naming the exact field violation plus a correction example — this lets the model self-correct on the next turn instead of guessing blind. Fat tools with 15+ optional fields are themselves a reliability bug: decomposing into narrow tools with fewer than 10 fields measurably raises argument accuracy because the model has less surface to get wrong per call.

Idempotency has to live in the orchestration layer, not just the tool. A tool-level Idempotency-Key plus a dedup table is necessary but not sufficient: if the agent loop itself doesn't track "did I already attempt this logical step," it can generate three distinct tool-call IDs for what a human would call one action, and every layer downstream honors its own idempotency contract perfectly while the user still gets charged three times. The key must be derived from the logical operation — agent_run_id + step_id — not from the arguments, since a model that timed out and retried may not even reproduce byte-identical arguments the second time.

same logical retry, three tool-call IDs — where must dedup happen?

  agent loop times out on process_refund() → retries 3×

  tool_call_id:      A1            A2            A3
                      │             │             │
  key = args          ↓             ↓             ↓
                  Idem-Key:A1   Idem-Key:A2   Idem-Key:A3
                      │             │             │
                      ↓             ↓             ↓
     dedup table sees 3 distinct keys → all "new" → charged 3×

  ────────────────────────────────────────────────────────────

  key = agent_run_id + step_id  (same key on every retry)

                      A1            A2            A3
                      │             │             │
                      ↓             ↓             ↓
                  Idem-Key:R7-S4 Idem-Key:R7-S4 Idem-Key:R7-S4
                        └─────────────┼─────────────┘
                     dedup table sees 1 key → charged 1×

The numbers

Error classes worth separating

Transient (503, 429, network timeout) → retry with backoff. Structural (400, schema validation failure) → never retry automatically; surface to the model as a correction prompt. Partial success (email queued but send unconfirmed, record created but linked step failed) → the most dangerous class, because it looks like success in the trace and the model narrates it as done.

The core mistake is treating the prompt as the safety mechanism. Telling the model "don't loop more than 10 times" is a suggestion the model can ignore under context pressure; it isn't a control. The fix is a pre-call governor: a runtime function that runs synchronously before every model call and every tool call, checks a set of counters, and refuses to proceed once any counter is exceeded. This is the same shape as a rate limiter, not a prompt instruction.

In practice the governor tracks five independent budgets per run: step count (model-tool-model cycles), cumulative token usage (input+output across the whole run, not per-call), dollar cost, wall-clock time, and per-tool call counts for anything expensive or destructive (a paid search API, a `send_email`, a `delete_row`). A step budget alone is not enough — an agent that retrieves 4,200 log rows per call can blow the token budget in three steps while staying well under a 15-step cap. The budgets have to be checked as a set, and the check has to happen before the call is dispatched, not in a post-run log review.

The second mechanism is stagnation detection, for the case where the agent hasn't hit any budget but also isn't making progress — the classic "call `search_logs` with the same args, get results, decide it needs more context, call again" loop. The standard approach is to fingerprint each tool call as hash(tool_name + canonicalized_args) and count repeats. Three hash collisions on the same fingerprint is a common trigger for a forced stop or human handoff. On termination — whether by budget or stagnation — the agent should return a partial answer plus a machine-readable reason code (`step_budget_exceeded`, `stagnation_detected`, `cost_ceiling_hit`), and every gate decision should be written to a persistent ledger so a postmortem can reconstruct exactly which budget fired and when.

PRE-CALL GOVERNOR (sync) — runs before every model/tool call
checks 5 budgets as a set: steps · tokens · $ · wall-clock · tool-ct
                              │  gate: blocks dispatch if over
                              ▼
   ┌────────┐  tool call  ┌────────┐  tool call  ┌────────┐
   │ model  │────────────▶│  tool  │────────────▶│ model  │──▶ ...
   └────────┘             └───┬────┘             └────────┘
                              │
                fingerprint = hash(tool_name + canon_args)
                same fingerprint × 3  →  stagnation trip
                              │
        ┌──────────────────────┴─────────────────────┐
        │   any budget exceeded  OR  stagnation_hit    │
        └──────────────────────┬─────────────────────┘
                              ▼
     STOP → partial answer + reason_code → persistent ledger

The numbers

What actually happens without one

The failure mode isn't hypothetical: a four-agent LangChain pipeline with an Analyzer and a Verifier fell into mutual recursion over A2A — the Verifier kept asking for more detail, the Analyzer kept producing it — and ran for eleven days before anyone noticed an invoice for $47,000. Separately, a Claude Code recursion loop with no crash and no error burned $16,000-$50,000 in five hours doing exactly what it was told, indefinitely. In both cases logging and monitoring existed; nothing was watching synchronously.

Two failover layers solve different problems. Provider-layer failover keeps one model alive by rerouting the same model string to a different backend (e.g. Claude via Bedrock instead of the Anthropic API) when a provider throws 429s or times out; it's transient and usually automatic, as in OpenRouter's default provider routing. Model-layer fallback is the second line of defense: when every provider for the primary model is down, or the response fails moderation, or context overflows, you drop to a different model entirely, usually smaller and cheaper. Both need a capability matrix, not just a price ladder — a refund-authorization tool call requires a model tier that's allowed to hold write access; a summarization call doesn't. Gate tool availability per rung, not just per request.

The dangerous failure mode isn't the hard error, it's silent quality collapse: the fallback model returns a well-formed, confidently wrong answer and nothing in the pipeline flags it. Pair fallback with an eval gate on the degraded rung, not just a health check on latency. The second dangerous mode is the retry storm — every client backing off and retrying at the same intervals hammers an already-struggling provider harder. Circuit breakers fix this: trip to open after N consecutive failures, refuse calls outright for a cooldown window, then allow a single half-open probe request before resuming full traffic.

Don't wait for hard failures to trigger routing decisions. Run synthetic health probes — small, fixed 50-token completions fired every 20-30 seconds per provider/model pair — and track p95 latency against an SLO so a rung gets marked degraded and routed around before real traffic hits it. When the entire ladder is exhausted, the last rung is a product decision, not an engineering one: templated FAQ response, search-only retrieval mode, or a human handoff that preserves full conversation state so the user doesn't repeat themselves.

request
   │
   ▼
┌─────────────────────────────┐
│ Primary model, 1 provider    │◀── health probe (50-tok, 20-30s)
└───────────────┬─────────────┘
                │ 429 / 5xx / timeout
                ▼
┌─────────────────────────────┐
│ Provider failover             │  same model, other backend
│ e.g. Anthropic API → Bedrock  │  (automatic, transient)
└───────────────┬─────────────┘
                │ all providers down / moderation / ctx overflow
                ▼
┌─────────────────────────────┐
│ Model fallback                │  smaller/cheaper model
│  → eval gate checks answer    │  (opt-in config)
└───────────────┬─────────────┘
                │ eval gate fails, or breaker already open
                ▼
┌─────────────────────────────┐
│ Circuit breaker                │  N fails → cooldown →
│  closed → open → half-open    │  1 probe request → resume
└───────────────┬─────────────┘
                │ ladder exhausted
                ▼
┌─────────────────────────────┐
│ Product decision, not eng:    │  FAQ template / search-only /
│  degrade on purpose           │  human handoff, state kept
└─────────────────────────────┘

The numbers

Fallback path atrophy

A fallback path that isn't exercised by real traffic rots. An upstream context-object refactor or a config key rename can silently break the degraded branch while unit tests keep passing against stale fixtures. The only defense is chaos testing: force synthetic timeouts or 429s against a small percentage of production traffic on a schedule, so the degradation code path runs for real against current API contracts, not against a mock frozen the day it was written.

Ingestion uses recursive character splitting at 600-1000 tokens with 10-15% overlap for prose, or structural splitting that respects code blocks and table boundaries. The higher-leverage move is parent-child indexing: embed small chunks (a paragraph) for precise retrieval matching, but return the larger parent chunk to the LLM so it has enough surrounding context to answer correctly. Retrieval itself is hybrid: BM25 (sparse, lexical) and dense embeddings run in parallel, and their independently-ranked result lists are fused with Reciprocal Rank Fusion (RRF) rather than a weighted score blend, since RRF only needs rank position and is robust to the two systems producing wildly different score scales.

The single highest-ROI step is reranking. A cross-encoder reranker (Cohere Rerank 3.5, BGE-reranker) takes the top 50-100 hybrid candidates and re-scores each query-document pair jointly, which raw embedding cosine similarity cannot do because it never lets the query and document attend to each other. Only after reranking do you truncate to the top 3-5 chunks and hand them to the generation model, with an explicit instruction to cite sources and refuse when the retrieved context doesn't support an answer — this is what actually suppresses hallucination, not prompt wording.

Freshness is a separate problem from retrieval quality. The source system (docs repo, CMS, database) is the source of truth; the vector index is a derived, lossy cache of it. Deletes and edits in the source have to be propagated to the index via source-ID-keyed tombstones, and drift between source and index has to be caught by automated challenge sets, not assumed away.

 doc ── split (600-1000 tok, 10-15% overlap) ──► child chunk
                                                       │ embed
                                                       │ (→ parent)
                          ┌────────────────────────────┘
                          ▼
              ┌── BM25 (sparse) ──┐
   query ─────┤                   ├── RRF fuse (by rank, not score)
              └── dense (parallel)┘
                          │
                          ▼
              top 50-100 hybrid candidates
                          │
                          ▼
         cross-encoder rerank (query × doc, jointly)
                          │
                          ▼
              top 3-5 chunks ── swap child → parent chunk
                          │
                          ▼
         LLM generates: cite sources, refuse if
         retrieved context doesn't support an answer
    

The numbers

When the index and the model version diverge

An index holding vectors from two embedding model versions produces silently garbage similarity scores: cosine similarity assumes both vectors live in the same geometric space, and different model versions produce different spaces, so cross-version comparisons return numbers that look valid but aren't. Re-embedding a live index is not an in-place update — it's an offline migration: re-embed every chunk with the new model into a separate index, validate recall on the golden set, then cut retrieval over once it's ready, since a partial re-embed leaves old- and new-version vectors mixed in the same index. Each chunk needs a model-version field stored alongside the source ID it's already keyed by, so a version mismatch is caught by a check rather than surfacing as an unexplained recall drop.

Split evaluation by pipeline stage, not by eyeballing the final answer. Retrieval scoring is classical IR: Recall@k asks whether the chunk containing the answer made it into the top-k candidates at all; Precision@k asks how much of that context window is noise the model has to filter out. Neither needs an LLM judge — they're set-membership checks against a labeled golden chunk, so they're cheap, deterministic, and fast enough to run on every retriever or chunking change.

Generation scoring needs a judge because "did the model use the evidence correctly" isn't a set operation. Faithfulness (also called groundedness) checks whether each claim in the answer is entailed by the retrieved context — not by the model's pretraining. Answer relevance checks the answer actually addresses the question, independent of whether it's grounded. A response can score high on one and low on the other: perfectly faithful to a retrieved passage that doesn't answer the query, or a fluent, relevant answer that quietly pulls in a fact the context never stated.

Citation quality is a third, distinct axis on top of faithfulness: attribution. It isn't enough that a claim is supported somewhere in the retrieved set — the specific citation attached to that specific sentence has to be the span that supports it. Liu et al.'s audit of four production generative search engines (Bing Chat, NeevaAI, perplexity.ai, YouChat) found only 51.5% of generated sentences were fully supported by their citations and just 74.5% of citations actually backed their attached sentence — "citation theater," where the cited link is topically relevant but doesn't verify the claim.

        QUERY
          │
          ▼
   ┌───────────────┐
   │   RETRIEVER   │──▶ Recall@k     golden chunk in top-k?  >90%
   │               │──▶ Precision@k  noise ratio in window   >60-70%
   └───────┬───────┘
           │ top-k chunks
           ▼
   ┌───────────────┐
   │   GENERATOR   │──▶ Faithfulness      entailed by context?   >0.85
   │               │──▶ Answer relevance  answers the query?     >0.85
   └───────┬───────┘
           │ answer + citations
           ▼
   ┌───────────────┐
   │   CITATIONS   │──▶ Citation precision  span backs claim?  ~75%
   │               │──▶ Citation recall     claims fully supported? ~52%
   └───────────────┘

   TRAP: Recall@k fails (wrong chunk retrieved)
     └─▶ Generator stays faithful to THAT wrong chunk
         └─▶ Faithfulness score: high.  Answer: confidently wrong.

The numbers

Failure modes that don't show up offline

Distribution shift is the one a static golden set can't catch: a 200-500 query set built from last quarter's production logs will keep passing while 30% of live traffic drifts into a query shape you never sampled. That's why the golden set has to be versioned as code and refreshed from real, anonymized traffic, not synthetic generators, and why teams like Respan pair the offline suite with rolling faithfulness averages on live traffic rather than trusting a single CI run.

The infrastructure is boring on purpose: a versioned JSON array of 200-500 cases, a script that replays each case against the current prompt/model, a scorer that emits pass/fail, and a CI hook that blocks merge on regression. The part teams skip is stratification. A golden set built only from production traffic is blind to attacks; one built only from red-team prompts is blind to what users actually do. The working split is production sample (60%), adversarial (15%), hand-written edge cases (15%), and replays of past incidents (10%) — every shipped failure becomes a permanent regression case, not a one-off fix.

LLM-as-judge is a measurement instrument, not ground truth, and it needs the same calibration any instrument does. Zheng et al. (NeurIPS 2023) found GPT-4-as-judge agrees with human experts about 85% of the time on general chat tasks — but that number drops to 60-68% in specialist domains (legal, medical, code correctness) where the judge lacks the same expertise it's grading. The fix is mechanical: pull 50-100 cases, have a domain expert label them binary pass/fail with a one-line critique, run your judge prompt against the same cases, and compute agreement. Below ~85%, the rubric is too vague — collapse multi-dimensional Likert scales (1-5 helpfulness/accuracy/tone) into binary criteria, because Likert scoring from LLMs clusters near the top of the range and produces low-variance noise that can't detect a regression even when one exists.

Judge-model kinship is the failure mode that slips through calibration checks: if the model under test and the judge model share a lineage or training data, the judge over-rates outputs that "sound like" its own style rather than outputs that are actually correct. This is why serious pipelines (Anthropic's own release evals, Cursor's agent evals) still lean on deterministic scoring — exact match, regex, schema validation, code execution against test cases — for anything with checkable structure, and reserve the judge for genuinely open-ended quality where no other signal exists.

None of this replaces human review — it narrows where humans need to spend time. A judge can only score against a rubric someone wrote and only catch failure modes someone already thought to check for; humans are what discover the next failure mode, author (and periodically rewrite) the rubric itself, and adjudicate the cases where the judge and a deterministic metric disagree. In practice that's a standing cadence, not a one-time setup cost: a domain expert reviewing 20-50 fresh transcripts weekly, plus every judge/metric disagreement routed to a human within the same cycle — cheap next to a blended score that quietly drifts for a month before anyone notices.

GOLDEN SET (200-500 cases)
  60% prod sample | 15% adversarial | 15% hand-edge | 10% incident replay
        │
        ▼  replay vs current prompt/model
  SCORER: deterministic (exact match/regex/schema/code-exec) for
          checkable structure + LLM judge only for open-ended quality
        │
        ▼  score EACH dimension separately — never average them
  CI GATE: block merge on ≥1-2% regression, on the WORST dimension


JUDGE CALIBRATION (do this before trusting the judge above)
  50-100 cases ──► human labels (pass/fail + 1-line critique)
        │
        └──► same cases ──► judge ──► compare ──► agreement %
                                                        │
              ≥85% agreement          │        60-68% agreement
              (general chat)          │       (legal/medical/code)
              judge is trustworthy    │        rubric too vague:
                                      │        Likert 1-5 → binary

  watch for: judge shares lineage with model under test
             → over-rates outputs that "sound like itself"

The numbers

Minimal harness shape

{
  "id": "email_triage_urgent_invoice",
  "input": { "subject": "OVERDUE: invoice #4471", "body": "..." },
  "expect": { "label": "urgent", "must_not_contain": ["draft sent"] }
}

The unit of LLM observability is the span: a named, timed record of one operation — a generation call, a tool invocation, a retriever query — carrying inputs, outputs, and status. A trace links spans by parent-child edges into a tree rooted at the user's request, so a five-turn agent loop becomes a walkable structure instead of five indistinguishable log lines. The instrumentation layer that makes this vendor-agnostic is OpenTelemetry's GenAI semantic conventions: standard span attributes for model name, token counts, and finish reason, so a trace captured in one SDK reads the same in another backend. Auto-instrumentation covers the boring 80% — wrap the client, get spans for free — but agent branching, guardrail checks, and custom evaluators need manual spans, because auto-instrumentation only sees the calls it wraps, not the control flow around them.

Cost and latency live at the span level, not the request level. Token counts split into input, output, cached, and reasoning tokens, and only span-level attribution tells you whether a $1.40 request burned reasoning tokens on a judge call versus a cheap cache hit on a repeated prompt — Anthropic's prompt cache, for instance, has a 1024-token minimum cacheable block, so a short system prompt below that floor never gets the discount no matter how often it repeats. For streaming responses, the request's total duration is close to useless: a span can report 8.4 seconds, green, within SLO, while the actual user experience was four seconds of a repeated three-sentence loop before the connection dropped. What matters instead is time-to-first-token (TTFT) for perceived responsiveness, and the tail of per-chunk gaps (tail-TBT) for stalls that a mean duration averages away.

None of this is affordable to keep at 100% sample rate, so tail-based sampling — deciding what to retain after a trace completes, not before it starts — keeps every error, every anomalous-latency trace, every high-cost request, and every low eval-score span, while thinning the rest to a distributional sample. Attaching eval scores directly to spans is what turns tracing into a drift detector: the same dashboard that shows p95 latency can show a rolling eval-score average sliding down over a week, which is the first signal of a silent model or prompt regression that never throws an exception.

TRACE (root = one user request)
 |-- span: retriever.query ............ 120ms, 3 hits (2 duplicate)
 |-- span: llm.generate "judge" ........ $1.40, reasoning_tokens spent
 `-- span: llm.generate "answer" ....... duration 8.4s, status=stop
      |
      0s      TTFT     chunks stream in        stall       8.4s
      +--------o####################.............####o------+
               first tok   (tokens)      4s gap    last tok
      total span duration = 8.4s, green, no error attribute
      but 4s of that was a silent mid-stream stall (tail-TBT)
    

The numbers

Anthropic's console and OpenAI's usage dashboard both roll spend up by key or project, not by anything your product organizes around. The fix is a metadata object attached at the call site — feature, tenant_id, workflow_step, environment, and a root request_id that ties multi-step agent traces back to one logical unit of work — passed through as a request parameter (e.g. Anthropic's metadata.user_id field, or a header like X-TFY-METADATA on a gateway). Retrofitting this later doesn't recover the lost history: untagged spend from last quarter can never be reattributed, so the tagging discipline has to be mandatory at the harness layer before the first call ships, not bolted on once someone asks for a per-customer breakdown.

A gateway or proxy (LiteLLM, Helicone, TrueFoundry) is the practical place to enforce this, because it sits between every application and every provider and can reject untagged requests outright. The gateway computes cost per request against a versioned pricing table — model, input/output tokens, cache-read and cache-write tokens all priced separately — and never retroactively re-prices historical rows when the provider changes list prices, since that would silently rewrite last month's reports. Raw per-request logs are too fine-grained to query at dashboard latency, so the gateway also writes rollup rows (daily, grouped by team/feature/model/provider) into an aggregation store like ClickHouse or Postgres; reporting reads the rollup, not the trace store.

Reconciliation closes the loop: nightly, diff the gateway's computed total against the actual provider invoice. A gap almost always means an unmodeled cost class — retried calls after a timeout, prompt-cache write tokens (billed at a premium, not the cached-read discount), or extended-thinking tokens that some SDKs don't surface in the response object used for cost calculation. Once attribution is real, it becomes a routing input, not just a report: a tenant crossing a soft budget can be auto-downgraded from a frontier model to a cheaper one for the rest of the billing period instead of getting hard-cut off.

call site           gateway (LiteLLM/Helicone/TF)      stores
┌──────────┐  req  ┌──────────────────────────┐  raw  ┌──────────┐
│ feature  │──────>│ reject if untagged       │──────>│trace log │
│ tenant_id│       │ price(model, in/out tok, │       │(too fine │
│ req_id   │       │  cache-read, cache-write)│       │ to query)│
└──────────┘       └────────────┬─────────────┘       └──────────┘
                                 │ daily rollup
                                 ↓
                       ┌──────────────────────────┐
                       │ ClickHouse / Postgres    │<- dashboards
                       │ rollup: team/feature/    │   read THIS,
                       │ model/provider, by day   │   not raw log
                       └────────────┬─────────────┘
                                    │ nightly diff
                                    ↓
                       ┌──────────────────────────┐
                       │ actual provider invoice  │
                       └──────────────────────────┘
                       gap = retries, cache-write
                       premium, extended-thinking
    

The numbers

What breaks attribution silently

High-cardinality tags — a raw user_id instead of a hashed or bucketed tier — blow up the rollup table's row count and defeat the point of aggregating at all; tag the tier or plan, not the raw identity, unless per-user margin is the explicit product you're building. Confusing environment tags is the other common failure: dev and staging traffic sharing a key with prod means load-test runs show up as real customer spend on the invoice review.

The structural bug is that an LLM concatenates trusted instructions and untrusted data into one token stream with no protocol-level boundary between "what to do" and "what to process." A retrieved document, a tool result, or an email can carry text that reads as an instruction, and the model has no hardware-level way to mark it otherwise. This is why no single filter stops the class: the best published input classifiers top out around a 92% block rate, and naive prompt-level defenses like "do not reveal your system prompt" are consistently bypassed because the same model that's supposed to refuse is the one being manipulated.

The fix that actually holds is denying untrusted data the authority to act, not trying to spot malicious text. Object-capability gating treats each tool call as requiring an explicit, unforgeable authorization token rather than an if/else classifier decision; Gödel Labs' capability-transformer computes this as deterministic hard attention over a token matrix and reports dropping attack success on side-effect actions (send, delete, transfer) from 100% to 5.7% on the AgentDojo benchmark, while never blocking a legitimate task. The complementary architectural rule — sometimes called the lethal trifecta — is to never co-locate private-data access, untrusted-content ingestion, and an egress channel in a single agent context; splitting those across separate agents means a successful injection has nowhere to exfiltrate to.

Anthropic's own containment model for Claude Code and Cowork treats the agent itself as a potential insider threat and caps blast radius with environment-layer sandboxing (Firecracker microVMs, bubblewrap/Seatbelt) around anything that executes untrusted code, plus recipient-scope guards that stop a compromised agent from message-exfiltrating across tenants. Layered on top: an API-level instruction hierarchy (system role outranks user role outranks tool-output role) gives roughly an 80% reduction in direct injection on its own, a fast fine-tuned classifier (DeBERTa-scale, not an LLM) does first-pass input filtering, and strict output schemas force responses into typed JSON so an injected model has less room to return anything other than the expected shape. Slow LLM-judge review is reserved for async auditing of ambiguous cases, not the hot path — it's too costly to run on every turn and too slow to gate a live tool call.

BEFORE — lethal trifecta: one agent holds all three
  private-data access ────┐
  untrusted content ──────┼──► single agent ──► egress ──► exfil
  (web / email / doc) ────┘     reasons + acts    (send/post/call)

AFTER — split across agents, gated at the exit
┌───────────────┐   ┌───────────────┐   ┌──────────────────────┐
│  fetch agent  │   │   reasoning   │   │     egress agent     │
│   untrusted   │──►│     agent     │──►│      capability      │
│    content    │   │  private data │   │    token required    │
│   no egress   │   │   no egress   │   │  100%→5.7% attack    │
│   no secrets  │   │               │   │ success (AgentDojo)  │
└───────────────┘   └───────────────┘   └──────────────────────┘
    no single hop holds data + secret + a way out at once

The numbers

Why human review isn't the backstop

Human-in-the-loop approval is often pitched as the final safety net, but telemetry on real approval queues shows people approve roughly 93% of prompts they're shown, independent of actual risk — reviewers habituate to the interruption, not the content. That makes per-step human approval unsuitable as a primary control; the working pattern is to reserve human sign-off for a short list of genuinely irreversible actions (wire transfers, external sends, production deletes) and let deterministic capability gates handle everything else without asking a human at all.

Data leakage prevention

The two capture habits elsewhere in this lesson — attaching a metadata object at every call site and retaining every error and high-cost trace — mean the trace store now holds the same prompts and completions the containment rules above are protecting. Redaction has to happen before the request leaves your infrastructure, not after: run inputs through a deterministic PII scrubber (Microsoft's Presidio, or an equivalent NER-plus-regex pass for emails, phone numbers, SSNs, and card numbers) and replace matches with typed placeholders before the call is logged or sent to the provider, since anything already inside a third-party request body can't be un-sent. The trace store needs the same treatment applied to what's already landed: field-level redaction on the prompt/response columns, plus a hard retention TTL (30-90 days is typical for debug traces) rather than the indefinite retention that "keep every error" implies by default — a compliance request or breach turns unbounded trace history into unbounded exposure. On the provider side, confirm the setting explicitly rather than assuming it: Anthropic's and OpenAI's standard API traffic isn't used for training by default, but that default doesn't cover browser-based consumer chat products, and a written Zero Data Retention agreement — available from both vendors on enterprise plans — is the only way to stop the provider from retaining logs of the request at all.

Tenant isolation usually gets solved once, at the database, with a tenant_id column and a WHERE clause or Postgres Row-Level Security policy. Every other layer a request passes through — Redis, an in-process memory cache, a shared vector index, an LLM provider's prompt cache — sits outside that request-scoped middleware and has no native concept of tenant identity. If the cache key is built from a resource ID alone, e.g. project:1041, two tenants whose auto-incrementing IDs collide will silently read and write each other's cached rows. There's no error, no exception, no log line — just a cache hit that returns the wrong org's data. The fix is mechanical: every cache key must be constructed as {tenantId}:{resource}:{parameters}, with no code path allowed to build a key without it.

LLM prompt caching (Anthropic, OpenAI, Gemini) adds a subtler version of the same bug: a timing side-channel. The provider caches a hashed prefix of your request server-side and returns a cache hit faster than a miss — often 30-90% faster, at roughly 10% of the token cost. If two tenants share an account and one tenant's prompt prefix is now sitting in that account-level cache, a second tenant submitting the same or a similar prefix will observe a measurably faster response, which leaks the fact that a specific document, system prompt, or query existed. The mitigation is a per-tenant cache salt — an invisible prefix token unique to each tenant, injected before the cacheable prefix — so tenant A's cached entries can never be hit by tenant B's requests, even if the underlying content is identical. The salt must be stable per tenant, not regenerated per request like a timestamp or session ID: held constant across a tenant's own traffic, it partitions the prefix cache along tenant lines rather than invalidating it, so each tenant still gets full cache hits on their own repeated requests and the only thing lost is cross-tenant prefix sharing.

Vector indices have the same failure mode with worse optics: metadata filtering alone (WHERE tenant_id = ? applied after top-k retrieval) stops content from crossing tenant lines, but the retrieval latency pattern itself leaks whether nearby embeddings from other tenants exist in the shared space — geometry as a side channel. The workable architecture is a bridge: default every tenant into a shared index with native pre-filtering (not post-filtering) for the long tail, and automatically promote high-volume or regulated tenants to a physically separate index or namespace once they cross a size or compliance threshold.

          request --> auth middleware (tenant_id bound)
                             |
                             v
                   +----------------------+
                   |       Postgres       |  WHERE tenant_id=? / RLS
                   |     (audited 2x)     |  scoped, clean
                   +-----------+----------+
                              |  tenant_id context ends here
+------------------+------------------+
v                  v                  v
Redis/mem cache    LLM prompt cache   Vector index
----------------   ----------------   (shared index)
key=resource id    no per-tenant      post-topk
only, no tenant    cache-salt prefix  metadata filter
----------------   ----------------   ----------------
-> wrong tenant's  -> timing leak:    -> latency leaks
data served        reveals prefix     other tenants'
(no error logged)  existed already    embeddings exist

The numbers

Where the audit misses it

A documented case: a Flask API cached project responses in Redis keyed only by an auto-incrementing Postgres ID. The database layer had been audited twice and was clean — every query correctly scoped by tenant_id. The cache layer, sitting outside that audit's scope entirely, served one enterprise tenant's confidential project data to another for 72 hours before a support ticket ("wrong company name on my dashboard") surfaced it. Nothing in the stack threw an error.

The dispatch rule is mechanical, not philosophical: RAG is for knowledge that changes faster than your release cycle, fine-tuning is for behavior that must be consistent across every call (format, tone, jargon, refusal style), and prompting is for instructions you haven't yet proven need either. Prompting is frozen weights plus a system message — cheapest to change, weakest guarantee. RAG keeps weights frozen too but injects retrieved chunks into context at inference time, so the model reasons over documents, rows, or API results it never saw during training. Fine-tuning (in practice, LoRA or QLoRA adapters on top of a base model) edits the weights themselves, so the behavior is baked in and survives even when the prompt is short or sloppy.

The trap teams fall into is treating fine-tuning as a knowledge-injection tool. FineTuneBench (arXiv 2411.05059) fine-tuned GPT-4o mini on two knowledge sets: rephrased news facts scored 98%, technically dense code-API updates scored 6%, on the same commercial fine-tuning pipeline. The model isn't learning the fact, it's learning to complete a distributional pattern near the fact — that only holds up when the new information looks like text the model already understands well. Facts injected into weights decay in a way facts injected into a retrievable database do not: there's no query to re-run, no version to diff, no access control to apply. That's also why regulated industries default to RAG even when fine-tuning would be cheaper at inference — a vector store is auditable, a LoRA adapter's knowledge is not.

Distillation is the fourth lever and solves a different problem entirely: cost, not capability. You fine-tune a small 7B–13B model to imitate a larger frontier model's outputs on your specific task distribution, then route high-volume repetitive traffic to the cheap model. It only works after you have a large volume of consistent, narrow-task traffic to imitate — using it as a first move, before you know your task shape, just bakes in whatever mistakes the teacher model made.

 Failure observed in production
        │
        ├─ never saw this fact ─────────────► RAG
        │    (info changes faster than release cycle)
        │
        ├─ knew it, format/tone/jargon drifts ─► Fine-tune (LoRA/QLoRA)
        │    (must be consistent every call)
        │
        ├─ told once, forgot two turns later ─► Prompt / context fix
        │    (instruction unproven to need more)
        │
        └─ correct, but too pricey at volume ─► Distillation
             (task shape already known)

 trap: routing "never saw this fact" into fine-tune, not RAG —
 FineTuneBench: 98% rephrased facts vs 6% dense code-API facts,
 same pipeline, same model. Weights aren't auditable; a
 retrieval index is.
    

The numbers

Failure modes by technique

RAG's failure mode is retrieval noise: irrelevant chunks in context collapse accuracy, which is why production pipelines add hybrid (vector + keyword) search and reranking rather than trusting naive cosine similarity alone. Fine-tuning's failure mode is freshness decay: a fine-tuned model is a static snapshot that will confidently state obsolete facts with no mechanism to know they're stale. Long-context's failure mode is the lost-in-the-middle effect — stuffing a million-token window doesn't fix recall for information buried mid-prompt, it just moves the cost from a retrieval step to a token bill.

Continuous batching (the vLLM default) keeps the GPU saturated by interleaving requests at the iteration level instead of waiting for a whole batch to finish, but it only works because PagedAttention stopped treating the KV cache as one contiguous allocation per sequence — paging it like OS virtual memory eliminates the fragmentation that used to cap concurrent requests well below what the HBM budget allowed. Grouped-Query Attention shrinks that same KV cache further by sharing key/value heads across query heads, which is what lets a given GPU hold more concurrent sequences in the first place. None of these stack for free: FP8 quantization roughly doubles throughput per GPU with a small, model-dependent quality tax, and it's now the production default rather than the aggressive option FP16 shops treat it as.

Speculative decoding is the lever most likely to fool you in a benchmark. A small draft model proposes several tokens, the target model verifies them in one forward pass, and when the draft is right you've bought multiple tokens for the verification cost of one — a genuine 2-3x win at batch size 1, which is exactly how the papers measure it. At production batch sizes the win inverts: verifying draft tokens for a dozen concurrent sequences competes for the same compute the target model needs for its own decode step, and once acceptance rates vary per sequence you get batch-level stalls waiting on the slowest verification. Treat spec decode as a low-batch, latency-sensitive tool (voice, single-user chat), not a blanket throughput lever.

Prefill and decode have opposite arithmetic — prefill is compute-bound (TTFT), decode is memory-bandwidth-bound (ITL) — and colocating them on one GPU means a long prefill stalls every in-flight decode stream sharing it. Chunked prefill (interleaving prompt chunks with decode steps via max_num_batched_tokens) is the cheap fix; disaggregating prefill and decode onto separate GPU pools removes the interference entirely but taxes you a KV-cache transfer over NVLink/RDMA, and if that fabric is oversubscribed the transfer eats the latency budget you disaggregated to save.

COLOCATED — one GPU, prefill (TTFT, compute) vs decode (ITL, mem-bw)

  A  A  A ┌───PREFILL B───┐ A  A  A     A = decode token, stream A
  ITL ok  └── stalls A ───┘ ITL SPIKE   long prefill blocks decode

CHUNKED PREFILL — same GPU, prompt B sliced via max_num_batched_tokens

  A [b][b] A [b][b] A [b][b] A ...       [b] = one prefill chunk of B
  ITL stays steady, TTFT(B) stretches    cheap fix, no extra hardware

DISAGGREGATED — two GPU pools, KV cache moves between them

  Prefill: ┌──PREFILL B──┐
           └──────┬──────┘
                  │ KV-cache transfer (NVLink / RDMA)
                  ↓
  Decode:  A  A  A  A  B  A  A  A        B joins, no stall...
                  └─ unless fabric oversubscribed →
                     transfer tax eats the saved latency

The numbers

Goodput over throughput

Raw GPU utilization is a vanity metric — a GPU pegged at 100% while every request blows its latency SLO is a worse system than one at 70% that meets it. Goodput (tokens served within the SLO) is the number that actually maps to revenue and $/token, which is why admission control — rejecting requests at the gateway that will already miss their SLO — improves outcomes for everyone else instead of letting them queue-clog the batch. Autoscalers should key off queue depth and KV-cache occupancy, not GPU utilization, because a memory-bound decode-heavy workload can show low compute utilization while still being fully saturated.

The dominant failure mode is a fail-plausible hallucination: the model gets a bad signal from a tool (an HTTP 400, an empty retrieval, a timeout) and instead of surfacing the error, synthesizes a fluent, confident response that hides it — an insight digest built on a failed API call reads identically to one built on real data. This is a downstream effect, not a model-quality problem: production tracing consistently shows the model is doing exactly what its context told it to do, and the context was wrong.

Tool-call errors form the second layer, and they split at the argument boundary. Loose schemas let the model return a freeform sentence where an enum was expected, or omit a field the schema marked optional but the downstream API required — these look like normal 200s until the API 500s three calls later. The worse case is partial execution: the tool call is accepted, the API returns 200, but the state mutation never commits (a write that raced a lock, a webhook that fired before the DB flush). Nothing in the trace looks wrong, so the agent proceeds as if the action succeeded.

Reliability compounds multiplicatively across steps, which is why single-step accuracy numbers are misleading: a 10-step workflow at 99% per-step reliability nets roughly 90% end-to-end success, and each additional tool call multiplies the failure surface rather than adding to it. Runaway execution (unbounded reasoning loops, context bloat inflating cost per turn) and eval-prod skew (a static test set that passed in January silently drifting out of distribution by March) round out the taxonomy — both invisible without continuous, trace-based evaluation against real production inputs.

                    tool call
                        │
                        ▼
              ┌─────────────────────┐
              │     response back     │
              └──────────┬────────────┘
           ┌─────────────┴──────────────┐
       200 OK                      error (400/timeout/empty)
           │                              │
           ▼                              ▼
  state mutated as              model synthesizes a fluent,
  intended?                     confident answer over the gap:
     │      │                  "fail-plausible hallucination"
  yes│      │no                              │
     ▼      ▼                                │
   done   partial execution                   │
          (write raced/lost —                 │
           trace still reads 200)              │
             │                                 │
             └───────────────┬─────────────────┘
                             ▼
                   nothing alerts either way
                             │
                             ▼
        compounds per step: .99 × 10 steps ≈ .90 end-to-end
    

The numbers

Typed error recovery

Not every failure deserves the same response: transient errors (rate limits, timeouts) should retry with backoff, invalid-input errors should route back to the model with the validator's message as context so it can self-correct, and fatal errors (auth revoked, resource deleted) should halt the agent rather than retry into a storm. Collapsing these into one generic "retry on error" handler is how a single expired token turns into a cascading incident.