A request asks a language model to plan a train trip. The model writes its answer one small piece of text, or token, at a time. Each new token needs information saved from earlier tokens. On the paper's OPT-13B configuration, model weights already occupy 26 GB of a 40 GB NVIDIA A100. The growing record competes for much of the remaining space.
Our train-trip prompt is an illustration, not a recorded benchmark request. Imagine a second request arriving while the first answer is still growing. The question is whether both records fit in memory, so the GPU can serve them together.
A cache that grows with every answer
In an attention layer, a query asks which earlier positions matter. Each earlier position has a key, which helps score that match, and a value, whose information contributes to the result. Each key and value is a vector, a list of numbers. Recomputing every older vector at every generation step would repeat work, so the server stores them in a key-value cache, or KV cache. The cache has a pair for each position across the model's layers. The same written token at another position may have a different cached state.
The paper works out the bill for OPT-13B: two vectors per token, 5,120 numbers per vector, 40 layers and two bytes per number. That is 819,200 bytes, about 800 KB per token. At a sequence length of 2,048, a single request can need about 1.6 GB of KV storage. These are numbers for the paper's model, not a universal size for language models.
The prompt phase can work on known input tokens together. During answer generation, each output token depends on the one before it. The paper studies how to keep more of those live requests in a batch by using the available GPU memory more effectively.
