What it is
When a large language model generates text, it produces one token at a time, and each new token must be compared against every token that came before it — including the ones the model itself just wrote. Without any optimisation, that comparison work grows quadratically as a reply gets longer: generating the tenth word requires redoing almost all the attention calculations for the first nine.
Key-value caching solves this by storing the intermediate “key” and “value” vectors — numerical representations the model computes for each token inside its attention layers — so they do not have to be recomputed on every subsequent step. Generation happens in two phases: a prefill phase that processes the full prompt at once and fills the cache, and a decode phase that computes keys and values only for the newest token and reads everything else back from memory. The result is that a model answering token 500 does roughly the same amount of fresh work as one answering token 5, rather than 100 times more.
The cache is not free. Its size grows with every token of context, and in long-context or high-volume serving it can consume more GPU memory than the model’s own weights. That has made cache size and management — through techniques such as grouped-query attention, quantisation, and DeepSeek’s own “multi-head latent attention” (which the company says cuts cache size by over 90 percent) — one of the most consequential engineering problems in deploying large models at scale, rather than a settled, interchangeable detail.
Why it matters for AI governance and narratives
The observatory’s interest in this term is not technical curiosity; it is about what counts as “open” in open-source AI. DeepSeek has built part of its international standing on the claim that it offers genuine openness as a counterweight to closed American labs. On 20 October 2026 the company released DeepInfer, the inference engine behind its production services, under an Apache 2.0 licence — a real and permissive release, covering the engine, its kernels, and its scheduler for the DeepSeek-V4.1-Flash model.
But DeepSeek’s own release notes, as read by the French outlet ActuIA, state that KV-cache handling and load-balancing between server replicas remain “tightly coupled to our internal infrastructure” and are “not yet fully open.” That is precisely the kind of distinction the observatory tracks under its symmetric-skepticism principle: a genuinely open licence that nonetheless withholds the specific engineering layer that determines how efficiently — and cheaply — a model can actually be run at scale. The framing contest here is not abstract. “Open-source AI” is a term different ecosystems use to claim legitimacy, and the gap between the licence text and the operational reality of what can be self-hosted competitively is exactly where that claim is tested.
Key facts and dates
DeepSeek had previously declined to open-source its inference engine in April 2025, citing an aging internal fork of the vLLM framework, tight coupling to its own infrastructure, and limited engineering capacity to maintain a public release. The October 2026 publication of DeepInfer reverses that decision only partially: the released code supports offline inference for DeepSeek-V4.1-Flash (a 552-billion-parameter mixture-of-experts model released 10 September 2026) on Nvidia SM100 GPUs and Huawei Ascend 950 NPUs, but DeepSeek has stated that KV-cache management, cross-replica load balancing, and full separation of the prefill and decode phases are excluded because they remain bound to internal systems.
Separately, DeepSeek’s base model weights (R1, V3, V4) have circulated under an MIT licence, and training and inference code has in other instances been released under Apache 2.0 — the same licence family used for DeepInfer. The technique of key-value caching itself is not DeepSeek’s invention; it is a standard feature of transformer-architecture inference used across essentially every major LLM deployment, from Llama to GPT-class models, which is what makes DeepSeek’s choice to carve it out of an otherwise open release notable rather than routine.