Understanding Compounding Context Costs in Agentic Loops

Introduction

As software engineering transitions from single-turn Large Language Model (LLM) completions to autonomous, multi-agent frameworks (such as LangGraph, CrewAI, and AutoGPT), developers are encountering a hidden operational hurdle: compounding context costs.

When an agent executes a multi-step task—such as a ReAct (Reasoning + Acting) loop—it does not call the model with isolated prompts. Instead, it must maintain a running transcript of its thoughts, function selections, and tool responses. Because LLM APIs are stateless, every iteration requires sending the entire historical context back to the model. This results in quadratic token scaling ( $O(N^2)$ ), driving up API bills, exhausting context limits, and degrading latency.

flowchart LR
    A["Step i prompt = base prompt + full history"] --> B["Model emits thought + tool call"]
    B --> C["Tool executes, returns observation"]
    C --> D["Observation appended to history"]
    D -->|"next step: i = i + 1"| A

Why the loop is expensive: history never shrinks, so step i resends everything from steps 1 through i-1 — the resend, not the new work, is what grows quadratically.

The Mathematics of Quadratic Context Growth

To understand the cost trajectory, let us model the token processing mathematically.

In a standard single-turn completion, the cost is simple: $\text{Cost} = \text{Input Tokens} \times \text{Input Price} + \text{Output Tokens} \times \text{Output Price}$

In an agentic loop, let us define:

$P_{\text{base}}$ : The static base prompt (system guidelines, guidelines, and schemas).
$T_{\text{out}}$ : The average tokens generated by the model per step (the “thought” and the tool call parameters).
$T_{\text{obs}}$ : The average tokens returned as tool observations (e.g. database output or webpage content).

For each step $i$ (where $1 \le i \le N$ ), the prompt sent to the model contains the static base prompt plus the history of all previous steps: $\text{Input Tokens}_i = P_{\text{base}} + (i - 1) \times (T_{\text{out}} + T_{\text{obs}})$

The total number of input tokens processed across a loop of $N$ steps is the sum of inputs at each step: $\text{Total Input Tokens} = \sum_{i=1}^{N} \left[ P_{\text{base}} + (i - 1)(T_{\text{out}} + T_{\text{obs}}) \right]$ $\text{Total Input Tokens} = N \cdot P_{\text{base}} + \frac{N(N - 1)}{2} \cdot (T_{\text{out}} + T_{\text{obs}})$

This equation reveals the quadratic term $\frac{N(N - 1)}{2}$ . As the number of steps $N$ increases, the input tokens grow quadratically. For instance, in a 10-step loop, the system processes the static base input 10 times and accumulates 45 times the average single-turn thought/observation history.

Prefill vs. Decoding: The Latency Impact

This quadratic token growth directly impacts execution speed due to how GPUs process transformer attention layers:

The Prefill Phase

Before generating new tokens, the serving engine processes the entire input prompt in parallel, computing Key-Value (KV) matrices. While GPUs perform this matrix multiplication in parallel (utilizing Tensor Cores), processing a massive accumulated context still takes significant time (known as Prefill Latency).

The Decoding Phase

Tokens are generated one-by-one auto-regressively. Each new token requires reading the KV matrices of all previous tokens. If the context has grown to 50k tokens, the GPU must fetch these matrices from memory for every single token generated. This is highly memory-bandwidth bound, leading to a drop in TPS (Tokens Per Second).

Strategies for Context Optimization

To deploy agentic architectures at production scale, engineers must implement active context management techniques:

1. Sliding Window Attention & Truncation

Instead of keeping the entire chat history, retain only the last $K$ steps of detailed logs. Older steps are discarded or truncated, capping the maximum input size to prevent it from growing indefinitely.

2. Summary Summarization

When older thoughts and tool outputs are truncated, pass them through a lightweight summarizer model (like a fast small language model) to condense the historical highlights into a single compact summary block, injected directly under the system prompt.

3. Selective Schema Loading

Instead of loading all 50 database tool schemas in the base prompt at step 1, use a vector search index to dynamically retrieve and inject only the 3 schemas most relevant to the current task step.

When hosting models using serving engines like vLLM or Triton Inference Server, enable Prefix Caching. Since the base prompt ( $P_{\text{base}}$ ) remains identical across all steps, the engine caches the key-value tensors for this prefix, skipping the redundant prefill computation on subsequent turns.