← 05 / Writing
2026-09-21/local-ai6 min readLIVE

Prefill vs. Decode

Prefill, or prompt processing (PP), reads the input. Decode, or token generation (TG), writes the output. GPU compute matters for ingesting repository context; memory bandwidth helps explain generation speed.

A local coding agent can stream an answer quickly and still feel slow every time it reads more of a repository.

The reason is that an LLM does two different jobs: it processes the input, then generates the output. Those jobs are prefill and decode, also called prompt processing (PP) and token generation (TG).

For agentic coding, that distinction matters every time the model receives source files, search results, or terminal output. The speed of the answer appearing on screen tells only part of the story.

Prefill is PP. Decode is TG.

PhaseAlso calledWhat the model doesMain performance pressure
PrefillPrompt processing (PP)Processes the input tokens and builds reusable attention stateCompute throughput and efficient matrix/attention kernels
DecodeToken generation (TG)Generates new tokens sequentiallyOften memory bandwidth for one active sequence

These abbreviations also appear in local inference tools. For example, llama.cpp uses pp for prompt processing and tg for text generation. Here, TG means the generation of output tokens, including reasoning and tool calls.

A request looks roughly like this:

Input prompt → PP → First output token → TG → More output tokens

PP tokens/s measures how quickly the model processes input. TG tokens/s measures how quickly it produces output. They describe different work, even though both use tokens per second.

Why PP benefits from GPU compute

During prompt processing, the input already exists. The model can process many token positions together, turning much of the work into large matrix multiplications. Causal attention still prevents a token from attending to future tokens, but it does not require the prompt to be processed one token at a time.

Those large operations give a GPU enough parallel work to use its compute units efficiently. Reusing the same weights across many input tokens also increases the amount of math performed per byte fetched from memory. This is why substantial PP workloads tend to benefit from compute throughput and efficient matrix/attention kernels. NVIDIA explains this difference through the shapes of the prefill and decode operations.

A capable GPU is usually the better choice for these large matrix operations, provided the model fits in accelerator-accessible memory and the runtime supports it well. CPUs can perform the same calculations, but usually offer much less parallel matrix throughput. Apple Silicon's integrated GPU can do this work from unified memory too.

The software matters alongside the hardware. A kernel is the routine that executes an operation on the processor. Optimized matrix kernels, fusion, and attention implementations determine how much of the hardware's capability becomes useful PP throughput. FlashAttention, for example, reduces attention's memory traffic by organizing the computation around the GPU's memory hierarchy.

For a rough timing example:

PP time ≈ uncached input tokens / PP tokens per second

40,000 tokens / 2,000 PP tokens/s ≈ 20 seconds
40,000 tokens /   500 PP tokens/s ≈ 80 seconds

These are illustrative rates. Actual PP throughput changes with prompt length, model architecture, and runtime. That processing time contributes to time to first token (TTFT), along with tokenization, scheduling, and other overhead.

Why TG often depends on memory bandwidth

During ordinary autoregressive generation, the next token depends on the tokens already generated. For one conversation, the model cannot simply process the entire future answer in parallel.

Each generation step still requires matrix operations. But with just one active sequence, those operations have much less work to reuse the loaded weights across. A dense model too large for the processor's on-chip cache typically has to stream most of its weights from memory again for each token. Attention also reads cached state. NVIDIA describes this as a common memory-bound inference workload.

That makes memory bandwidth a useful first approximation for TG speed:

ideal TG tokens/s ≈ memory bandwidth in GB/s / model weight size in GB

The denominator is the weight size in bytes at the precision being used. A 27B model has about 27 billion parameters; that does not mean its weights occupy 27 GB. Quantization changes how many bytes represent those parameters.

This estimate assumes one full weight read per generated token at peak bandwidth. It ignores additional memory traffic and execution costs, so it is an optimistic estimate for ordinary batch-one dense-model decode.

M5 Ultra and Qwen3.8-27B: a worked example

Apple lists 1.2 TB/s of memory bandwidth for the Mac Studio with M5 Ultra. Using decimal units, that is about 1,200 GB/s.

Qwen3.8-27B is a dense model. Unsloth lists its UD-Q4_K_XL GGUF at about 17.6 GB. Taking that as an approximate weight payload gives:

ideal TG ≈ 1,200 GB/s / 17.6 GB
         ≈ 68 tokens/s

That 68 tokens/s is a calculation, not a measured M5 Ultra result. It assumes the model is resident in memory and inference achieves all the advertised bandwidth. Real execution also pays for cached attention state, activations, quantization unpacking, kernel launches, and sampling. The file size is only an approximation for the bytes actually streamed.

For comparison, the same calculation at 600 GB/s gives about 34 tokens/s. That shows why doubling bandwidth can help ordinary decode even when peak compute is not the limiting factor.

The estimate applies to one active sequence without speculative decoding. Batching can reuse weights across sequences; speculative decoding can accept multiple output tokens from one target-model pass. Both change the assumptions. Longer occupied context also adds attention work and memory traffic, making a weights-only estimate less representative.

Why PP matters so much in an agentic coding harness

A coding harness runs a loop around the model: it supplies instructions and context, executes tool calls, and returns the results for the model to process.

That loop might look like this:

Repository map + instructions → PP → TG: request source files
Source file contents         → PP → TG: request related files
More files + search results  → PP → TG: propose an edit
Test output + diff           → PP → TG: diagnose and revise

The tool reads bytes from disk. The model then processes the returned text to reason about the code. A fast filesystem does not remove that PP cost.

The input can be much larger than the generated tool call or patch. An agent might ingest tens of thousands of tokens across several files before generating a few hundred tokens to choose its next action. For repository exploration and context-heavy tool loops, PP can dominate the time spent waiting.

Consider two hypothetical systems processing 40,000 uncached input tokens and generating 500 output tokens:

SystemPP speedTG speedPP timeTG timeCombined model time
A500 tokens/s80 tokens/s80 s6.25 s86.25 s
B2,000 tokens/s40 tokens/s20 s12.5 s32.5 s

These figures exclude tool execution and other overhead. System B finishes this interaction much sooner because it processes the repository context faster. System A's higher generation speed saves only a few seconds after a much longer wait.

Generated reasoning, tool-call arguments, and patches all count toward TG. A long reasoning run can still spend most of its time generating tokens. The balance depends on what the agent is doing.