The Lifecycle of an LLM Prompt: From Input to Output

From Raw Text to Token Sequences

The execution lifecycle of a Large Language Model prompt spans two distinct operating domains: raw string manipulation and matrix mathematical operations. The path from the user’s keystrokes to the streaming generated output follows a highly optimized pipeline.

graph TD
    Keystrokes[Raw User Text] --> Tokenizer[Byte-Pair Encoding Tokenization]
    Tokenizer --> Prefill[Prefill Phase: Parallel Key-Value Projections]
    Prefill --> KVCache[KV Cache Allocation]
    KVCache --> Decode[Decode Loop: Autoregressive Token Generation]
    Decode --> Detokenizer[Detokenizer: Token ID back to String]
    Detokenizer --> Stream[HTTP SSE Streaming Output]

---

1. Tokenization and Embedding (Ingress)

When a prompt reaches the inference server, it cannot be processed as raw characters. It must first pass through a Tokenizer (typically using algorithms like Byte-Pair Encoding or SentencePiece):

• BPE Mapping: The input string splits into discrete tokens (words, sub-words, or punctuation coordinates) and maps to unique integer IDs in the model’s vocabulary list.
• Vector Projection: Each token ID is matched against a lookup weight matrix to retrieve a dense input vector of size $d_{\text{model}}$ (e.g. 4096 dimensions).
• Positional Injection: Positional coordinates are added (via sine/cosine vectors or Rotary Position Embeddings) to encode word order directly into the vectors.

---

2. The Prefill Phase (Parallel Processing)

The model processes all input prompt tokens simultaneously to initialize the generation state:

• Matrix Parallelism: The token vectors pass through the transformer’s multi-layer self-attention blocks in a single, parallel compute pass.
• KV Cache Creation: The computed Key ($K$) and Value ($V$) vectors for each input token are saved into a dedicated GPU memory area. Caching prevents the server from recalculating these representations on subsequent turns, saving massive floating-point operations.

---

3. The Decode Phase (Autoregressive Loop)

Once the prefill phase finishes, the model switches to token-by-token generation. Since attention is autoregressive, each new token depends on all previous tokens:

1. Next Token Prediction: The model calculates self-attention for the current token using the pre-existing KV Cache.
2. Logit Evaluation: The output of the final transformer layer yields a vector of raw scores (logits) across the entire vocabulary space.
3. Sampling Decision: The logits pass through softmax normalization modified by hyperparameters (Temperature, Top-P, Top-K) to select the next single token ID.
4. Cache Appending: The Key-Value vectors for the newly generated token are appended directly to the KV Cache.
5. Iteration Loop: Steps 1 to 4 repeat continuously until an End-of-Sequence (<EOS>) token is generated or the maximum length target is hit.

---

4. Detokenization and Streaming (Egress)

As each token ID is generated in the decode loop, it is passed to a detokenizer to convert the integer back into readable text. The server streams these characters immediately back to the client using Server-Sent Events (SSE), ensuring low perceived latency for end users.