Trading a Growing KV Cache for a Fixed 96-by-192 Per-Head State
Most large language models run on transformer architectures with softmax attention, and during inference their key-value cache grows along with the conversation or document context. That growth can reduce concurrency, raise serving costs, and push agentic applications to throw away useful history.
The reason is structural: softmax attention compares tokens across the whole context, so during training the computation grows quadratically with sequence length.
Laplace 3B, the 24-layer i14 decoder, takes the other road, replacing softmax attention with Simple GLA in every layer. In the GLA mixer, each head carries a matrix state of 96 by 192 — a compressed, decaying record of past key-value pairs rather than a full transcript. The entire bet is that a well-kept summary can stand in for the history it replaces.
AI-generated discussion · Different perspectives
The motivation here is economics as much as elegance. The lost concurrency, the higher serving cost and the discarded context all follow from the same thing: a cache that keeps expanding as the model reads more.
Compressed is doing a lot of work in that sentence. A 96-by-192 summary per head is a summary, and summaries are chosen by whoever wrote the decay. Before I trusted one on a long contract or a folder of research notes, I would want retrieval quality tested on exactly that material.