How LLM Inference Works: Attention, KV Cache and Reasoning

One request, several engineering problems

A language model can produce a useful answer in a product demo and still be unsuitable for the same task under production load. Input processing, generation speed, memory consumption and answer quality are separate constraints. Understanding the inference path makes it easier to diagnose which constraint is responsible.

This article follows a decoder-only, autoregressive Transformer: the architecture behind many deployed text models. Multimodal encoders, mixture-of-experts layers and alternative sequence architectures introduce additional details. The calculations below are explicit examples, not measurements from a Nikba deployment.

From a conversation to the next-token distribution

The application serialises instructions, conversation turns, retrieved documents and tool results into the model's expected input format. A tokenizer maps that representation to token IDs. Token boundaries depend on the tokenizer; a word, a code fragment or a non-English phrase can occupy several tokens.

The model combines token representations with position information and transforms them through successive layers. In causal self-attention, a position can use earlier positions and itself, while future positions are masked. Query, key and value projections determine how information from accessible positions contributes to a representation.

Attention(Q, K, V) = softmax(QKᵀ / √dₖ + causal_mask)V

The final representation produces vocabulary scores, or logits. A decoding rule converts those scores into a selected next token. Training has already adjusted the model's parameters; ordinary inference uses those parameters without updating them. The original Transformer paper explains the attention mechanism, although its full architecture includes both an encoder and a decoder.

Prefill and decoding have different bottlenecks

During prefill, the server processes the known input sequence and prepares the state needed for generation. It can parallelise substantial computation across input positions. During decoding, each newly generated token extends the sequence and conditions the next step. For conventional autoregressive generation, an answer with many output tokens requires many successive decoding steps.

Consequently, time to first token and output tokens per second describe different parts of the experience. Queueing, retrieval, network transport and application work also contribute to user-visible latency. A fast decoder does not compensate for an overloaded request queue. NVIDIA's inference analysis describes the compute and memory trade-offs behind these phases.

For a hypothetical document assistant, measure retrieval separately from model prefill. If short and long answers both start slowly, inspect the queue, retrieval and input length. If the first token arrives promptly but completion is slow, inspect output length, decoding throughput and concurrency. These are diagnostic hypotheses to test against traces.

The KV cache trades repeated computation for memory

Autoregressive serving can retain earlier keys and values instead of recalculating them at every decoding step. The KV cache is an inference optimisation. It is not a record of verified facts, a database of company knowledge or a substitute for the model's context.

For a full-attention model, a useful first estimate of cache storage for one sequence is:

KV bytes ≈ 2 × layers × cached_tokens × kv_heads × head_dimension × bytes_per_element

Example assumptions:
32 layers × 32,768 tokens × 8 KV heads × 128 dimensions
2 arrays (K and V), each using 2 bytes per element

2 × 32 × 32,768 × 8 × 128 × 2 = 4,294,967,296 bytes ≈ 4 GiB

This estimate excludes model weights, activations, allocation overhead and other requests. Different attention patterns, cache formats and quantisation change the result. It explains why a model that fits in accelerator memory can still fail when several long conversations arrive together. Hugging Face documents cache strategies with different memory, latency and compilation characteristics; the best choice depends on the serving workload.

Decoding controls variation, not factual authority

Greedy decoding selects a highest-scoring token at each step. Sampling introduces variation; temperature and candidate filtering alter the distribution used for that selection. A low temperature does not turn a plausible statement into a verified one. A repeatable answer can still contain a factual error.

Keep the model version, prompt format and decoding settings fixed while comparing application changes. Use an appropriate output budget and stopping conditions. See the generation documentation for supported decoding strategies rather than assuming that every provider exposes identical controls.

Speculative decoding can propose multiple tokens and verify them with a target model. Distribution-preserving algorithms can accelerate generation without changing the intended target distribution, but speed depends on proposal acceptance and serving overhead. It is not a universal multiplier. The speculative decoding paper provides the underlying method.

Reasoning introduces another inference budget

Some reasoning models are trained to use extended intermediate generation before delivering an answer. DeepSeek-R1 is a documented example of reinforcement learning applied to reasoning behaviour. That result does not establish that every task benefits from longer deliberation, or that a generated explanation faithfully exposes the model's internal computation.

For a product, compare reasoning budgets on representative tasks. A constrained classification step and a multi-stage planning problem may need different settings. Measure additional latency and cost against the rate of correct, usable outcomes. Include cases where the system should request missing information or decline to make an unsupported claim.

A measurement plan before choosing a larger model

Build a small, versioned workload containing short requests, long documents, code, multilingual input and the expected tool outputs. Test it at realistic concurrency. Record queue time, input and output tokens, time to first token, completion latency, memory pressure and task success. Inspect p50 and p95 latency so that averages do not hide a poor experience for slower requests.

Change one factor at a time: model, context length, retrieved material, decoding settings or serving configuration. An illustrative acceptance criterion might require a correct answer with a valid source before counting the request as successful. Set its numerical threshold from the product's needs and baseline measurements, not from a generic tokens-per-second target.

For the next architectural decision, continue with RAG versus fine-tuning. For systems that execute actions, see reliable AI agents.