LLM Request Lifecycle
One request, two very different phases
A language-model request begins like an ordinary API call: authenticate, apply rate limits, assemble the prompt, and send it to a provider. Inside the model, however, the work splits into prefill and decode.
During prefill the model reads the input context. Those tokens can be processed largely in parallel, so a longer prompt raises time to first token without adding one equal delay per token. During decode the model produces output one token at a time. Token 200 cannot exist before token 199, which is why output length has such a direct relationship with latency.
This gives an LLM application two latency numbers. Time to first token (TTFT) measures how long the blank screen lasts. Full completion latency measures when all work is actually finished. Streaming improves the first number the user experiences; it does not make decoding finish sooner.
client → gateway → prompt assembly → model
│
prefill │ reads input in parallel
▼
first token ← TTFT
│
decode │ one output token at a time
▼
final token ← completionOptimising TTFT and optimising completion latency are related but separate jobs. A streamed ten-second answer can feel responsive while still occupying provider capacity for ten seconds.