Inference engineering in 2026: speculative decoding, batching and quantization

In LLM cost optimization I covered application-layer levers (model choice, caching, batching). If you serve your own models — or run high volume against one provider — the real wins live one layer down: the inference systems layer. 2026’s production measurements have clarified considerably what works there, and where it stops working.

Why decode is slow: diagnose first

In autoregressive generation each token is a separate step and the weights are re-read every step — decode is mostly memory-bandwidth-bound, not compute-bound. Prefill, by contrast, is compute-heavy. Optimizing both with the same strategy is a common mistake.

1. Speculative decoding

A small “draft” model proposes several tokens ahead; the large model verifies them in a single parallel pass. Since verification is cheaper than generation, several tokens fall out of each expensive pass. Formalized by Leviathan and colleagues in 2023, this became a production standard by 2026.

• Realistic gains: production benchmarks show 2-3x speedups at low-to-medium concurrency — but the gain compresses toward zero at high batch load where the GPU saturates. The decisive variable is not the algorithm, it is your concurrency tier (production benchmarks, 2026).
• Variants: EAGLE-3, DFlash, MTP and DSpark recover more headroom than first-generation implementations.
• Quality: the large model still verifies every token, so output quality is unchanged — that is the difference from quantization.

2. Continuous batching

With static batching a new request cannot start until the batch finishes; with continuous batching, new requests join as soon as a slot opens. It is the default in vLLM and TensorRT-LLM. Production reports show 2-3x throughput over static batching, and 3-5x for high-QPS background work (systems playbook, Cast AI). For most stacks this is the single highest-leverage change.

3. Quantization and KV cache

• Quantization (FP8/INT4): memory shrinks 2-4x and cost drops ~50%, but quality must be re-measured on your own workload — a loss invisible in summarization can be visible in code generation.
• PagedAttention / KV cache: vLLM-style paging cuts memory waste and manages the KV cache, the largest memory consumer in long contexts.

4. Do not stack blind

One telling data point: in Cast AI’s 2026 Kubernetes report, average GPU utilization across production fleets was 5%. Optimizations stack and their savings overlap; total-savings estimates without measuring each layer separately are misleading.

Order of operations

1. Diagnose the bottleneck (prefill? decode? memory?).
2. Continuous batching + right-sizing the model (biggest effect).
3. Speculative decoding (at low-to-medium load).
4. Quantization + KV cache tuning (with a quality eval).

For the application side see LLM cost optimization; for the context side see Context engineering.