Inside Kimi K3's Architecture: KDA, Attention Residuals, and the 1M Context That Actually Runs

Every headline about Moonshot AI’s Kimi K3 leads with the same number: 2.8 trillion parameters, the largest open-weight model released so far. That number is close to the least useful thing you can know about the model. Nobody activates 2.8 trillion parameters per token, the count tells you nothing about latency or quality, and “largest” has never correlated cleanly with “best.”

The part worth your attention is the attention itself. K3 makes a hybrid linear-attention stack its main mechanism rather than an appendix experiment, and at 2.8T parameters it is the largest model to do so. It is not the first to try the idea - MiniMax-01 shipped a 7:1 lightning-attention hybrid at 456B back in January 2025, and Nemotron and others have gone the Mamba route - but K3 pushes it to frontier scale with a tighter 3:1 ratio and a more expressive linear layer. The stack is Kimi Linear, the architecture Moonshot published in October 2025, scaled from a research paper into a production flagship. That is the reason a 1M-token context is economically usable here and not just a spec-sheet line.

This post is for engineers and architects deciding whether K3’s architecture actually changes their long-context or agentic workloads, or whether it is another leaderboard entry to note and move on from. I will walk the architecture layer by layer, then push back on two things the launch coverage gets wrong: the benchmark numbers, which are mostly Moonshot’s own, and the assumption that “open weights” means you can run it.

Why the attention stack is the story

Full softmax attention has a cost that every long-context deployment eventually hits. The KV cache grows linearly with sequence length, and at a million tokens the memory for that cache and the bandwidth to read it every decode step become the binding constraint, not the compute. You can serve a 1M context. Paying for it on every generated token is the problem.

Linear attention fixes the asymptotics by replacing the growing cache with a fixed-size recurrent state, but historically it gave up quality to do so, especially on recall-heavy tasks where a model needs to pull an exact fact from far back in the sequence. The interesting engineering question for the last two years has not been “can we do linear attention” but “can we do it without losing the thing full attention is good at.”

K3’s answer is not to pick one. It interleaves a strong linear-attention layer with periodic full-attention layers, and spends real design effort on making the linear layer expressive enough that the ratio can tilt heavily toward it. That is what the rest of this post unpacks.

The parameter count that matters is 104B, not 2.8T

Start with the number the headlines skip. K3 is a Mixture-of-Experts model with roughly 104 billion active parameters per token out of the 2.8 trillion total, activating 16 of 896 experts per token through what Moonshot calls its Stable LatentMoE framework. The MoE layout follows the Moonlight and DeepSeek-V3 lineage, so the sparse-routing design is familiar territory rather than a new bet.

The gap between 2.8T stored and 104B active is the whole point of sparse MoE, and it is why the total-parameter headline is misleading as a capability or cost signal. Each token only pays the compute of a 104B-active forward pass, an activation ratio near 1.8 percent (16 of 896) that is among the sparsest of any open model. For comparison, Kimi K2 ran around 1T total with roughly 32B active per token, so K3 more than doubled its expert count while pushing activation down, from about 2 percent to 1.8 percent. More capacity, not proportionally more active compute per token.

That sparsity is where the interesting systems problem starts. Your accelerator memory still has to hold the full 2.8T expert set, so the experts are physically spread across many GPUs, and the routing that picks 16 of them per token becomes traffic on the interconnect. The next section is how K3 keeps that from eating the efficiency the sparsity just bought.

Stable Latent MoE: routing 16 of 896 experts without drowning in GPU traffic

This is the part of the architecture that the launch coverage mostly skips, and it is the most interesting from a systems point of view. Sparsity looks free on a spec sheet and is anything but free on a rack of GPUs.

A model this large does not fit on one accelerator. At 4-bit MXFP4 precision, K3’s weights alone are roughly 1.4TB (2.8 trillion parameters at about half a byte each) before you store a single token of context, so the expert pool is spread across the GPUs of a full node or a supernode. Each token activates 16 experts that can live on different GPUs, which means the MoE layer fires an all-to-all exchange across the interconnect on every token. That traffic is not a rounding error. For a comparable EP64 configuration, all-to-all communication can consume roughly 20 percent of time when it stays inside an NVLink domain and 40 to 60 percent once it crosses node boundaries, per systems analyses of DeepSeek-V3-class deployments. Move the experts too far apart and the sparsity you paid for in compute comes back as network cost.

Stable Latent MoE attacks that at two points. First, routing happens in a compressed latent space rather than on the full token. The router projects each token down to a 3,584-dimension latent, half the 7,168 hidden size, before dispatching it to experts, so the vector crossing the interconnect and the per-expert compute are roughly halved, then projects back up before the layer output (RunPod’s K3 technical FAQ has the clearest write-up). Two shared experts run on every token alongside the 16 routed ones, absorbing the common patterns so the routed experts specialise.

The word “stable” is about the router, not the latents. Selecting 16 experts out of 896 is a brittle optimisation: a small change in the score distribution can swing which experts fire and starve the rest, and a poorly balanced router wastes capacity. The common fix, used by DeepSeek-V3 and Nemotron, nudges the router with bias terms that penalise overused experts and promote underused ones, which works but adds a sensitive balancing hyperparameter you have to tune. K3 uses quantile balancing instead: it derives expert allocation from the quantiles of the router-score distribution, grading on a curve rather than on fixed raw scores, and soft-drops overflow tokens rather than hard-bouncing them. The payoff is a router that stays balanced at 1.8 percent activation without the hyperparameter babysitting the bias-based schemes need.

Read the two ideas together and the intent is clear: latent routing keeps the per-token GPU-to-GPU traffic down, and quantile balancing keeps the router usable at an activation ratio that would destabilise a naive scheme. Both exist because the model is spread across a lot of silicon.

Kimi Delta Attention: the linear core

The linear-attention layer is Kimi Delta Attention (KDA). It extends Gated DeltaNet with a finer-grained gating mechanism, and the two words doing the work there are “delta” and “gating.”

The delta rule is a write correction. Instead of blindly accumulating information into the fixed recurrent state, KDA subtracts the stale value associated with a key before writing the fresh one. That keeps the finite state precise as it updates, which is exactly the weakness that made earlier linear-attention schemes forget or smear facts.

The gating is the refinement over Gated DeltaNet and Mamba2. The forget gate decays memory per channel rather than through one global scalar, so the layer can hold some dimensions of state stable while letting others decay fast. Practically, that finer control is what lets a fixed-size state behave more like selective memory and less like a leaky bucket.

The part that makes it viable on real hardware is a bespoke chunkwise algorithm built on a specialised variant of Diagonal-Plus-Low-Rank (DPLR) transition matrices. The general DPLR formulation is expensive; Moonshot’s specialised version cuts the computation substantially while staying closer to the classical delta rule. This matters because an expressive linear layer that is slow to compute buys you nothing - the whole reason to leave full attention is throughput.

The 3:1 hybrid, and why MLA drops positional encoding

KDA does not run alone. K3 interleaves KDA layers with Multi-Head Latent Attention (MLA) layers, the same full-attention mechanism DeepSeek-V3 uses, in a uniform 3:1 ratio: three KDA layers, then one MLA layer, repeated, each followed by an MoE feed-forward layer. Moonshot reports that 3:1 gave the best quality-throughput trade-off empirically, not from theory.

The division of labour is deliberate. The KDA layers carry the bulk of the sequence processing cheaply and hold recency and positional information in their recurrent state. The periodic MLA layers preserve global, exact-recall information flow that a purely recurrent stack degrades on. Three-quarters of the layers get linear-attention economics; one-quarter keeps the property full attention is actually good at.

One detail that trips people up: the MLA layers use NoPE, no position encoding. All positional and recency signal is delegated to the KDA layers, which removes the RoPE machinery from the full-attention path entirely. This is a genuine simplification rather than an omission - if the linear layers already encode position through their ordered state updates, making the full-attention layers position-agnostic avoids double-counting and drops a source of long-context extrapolation pain.

Attention Residuals: a cheap fix for deep networks

The one architectural addition K3 makes on top of Kimi Linear is Attention Residuals (AttnRes), documented in Moonshot’s own technical report. It is cheap in parameter terms but does more than the name suggests.

A standard residual connection adds each layer’s output back with a fixed unit weight, so depth-wise aggregation is the one part of a modern transformer that is not learned and input-dependent, unlike the attention and the expert routing around it. As depth grows, early-layer signal has to survive a long chain of fixed sums to reach the end, and hidden-state magnitudes drift. AttnRes replaces that fixed sum with a learned attention over earlier layer outputs: each layer weights which past layers it reads from. The parameter cost is one RMSNorm and one pseudo-query per layer, initialised to zero so the mechanism starts as an equal-weight average and does not destabilise early training.

At K3’s depth the naive version would be expensive, because letting every layer attend to every earlier layer is quadratic in depth and adds cross-GPU traffic. K3 uses the Block AttnRes variant, which groups the layers into a handful of compressed blocks (reported as 93 layers in 9 blocks) and attends at block granularity, cutting the memory and communication overhead. Moonshot reports multi-point gains from AttnRes on reasoning-heavy benchmarks when applied to Kimi Linear, so treat it as a cheap addition with real effect, not a rounding error. The efficiency headline still belongs to KDA and the 3:1 ratio; AttnRes is the piece that keeps a stack this deep trainable.

What the architecture buys

Put the pieces together and the numbers Moonshot reports for Kimi Linear, the architecture K3 inherits, are the reason any of this matters:

• Up to 75% reduction in KV cache usage during long-sequence generation, because three of every four layers hold a fixed state instead of a growing cache.
• Up to 6x higher decoding throughput at 1M context length compared to a full-attention MLA baseline.
• Roughly 2.5x overall scaling efficiency versus Kimi K2, which Moonshot attributes to the architecture together with training and data-recipe changes, not to attention alone.

Read those as a single claim: the 1M-token context on K3 is meant to be cheap enough to use inside a real agentic loop, not just large enough to advertise. For long-horizon coding agents that feed a repository, its tests, and its runtime logs into one context and iterate, the per-token decode cost at long context is the economic constraint, and that is precisely what the hybrid stack attacks. Whether it holds up under independent measurement at production load is the open question, and the honest answer today is that we do not have enough third-party data to confirm it.

The benchmarks: mostly Moonshot’s own, so discount accordingly

Here is where the launch coverage needs correcting. The coding numbers making the rounds - 67.5 on DeepSWE, 77.8 on ProgramBench, 88.3 on Terminal-Bench 2.1, 81.2 on FrontierSWE, 42.0 on SWE Marathon - are, as of late July 2026, mostly published by Moonshot itself, and several do not appear on the benchmark owners’ live leaderboards. (All figures Moonshot-reported) Each lab also tests its own model inside its own best-case tooling, which makes cross-model comparison unreliable even when the numbers are real.

There is some independent signal. Vals AI’s overall testing placed K3 second, below Anthropic’s Claude Fable 5 and above OpenAI’s GPT-5.6 Sol (Vals AI ranking as of July 2026, - leaderboard positions move weekly). Second place on an independent aggregate for an open-weight model is a genuinely strong result and the claim I would actually stand behind. It is a different and more defensible statement than “beats Fable 5 on coding,” which rests on Moonshot’s own tooling.

The architect’s takeaway: the current evidence supports a production pilot and a benchmark-aware evaluation on your own workload. It does not support a final verdict, and any post telling you K3 is now the best coding model is quoting a vendor’s self-report as if it were settled.

”Open weights” does not mean you can run it

The second correction is about deployment. Moonshot released K3’s weights under an open-weight license, and a lot of the excitement treats that as “so I can self-host it cheaply.” The hardware reality is blunt.

Running the full model needs a full high-memory GPU node, not a couple of cards. As noted earlier, the weights alone are roughly 1.4TB at 4-bit before any context, so day-0 serving guidance is at least one 8x B300 node or a GB300 NVL72 rack, with 16x B200 also supported (vLLM shipped day-0 support with these targets). Moonshot recommends supernodes of 64 or more accelerators for efficient serving. The reason is not only memory: because expert parallelism generates the all-to-all traffic described above, K3 wants a large high-bandwidth scale-up domain, an NVLink rack, so that expert routing stays fast. That raises the minimum viable serving unit from one GPU to a tightly coordinated supernode. (You will still see “8x H100” floors quoted in some coverage; treat those cautiously - 1.4TB of 4-bit weights does not fit in 8x 80GB, so a Hopper deployment needs far more cards or a smaller quantization.) The practical takeaway holds: this is not a model a small team spins up casually, and self-hosting does not automatically beat API pricing once you account for utilization, quantization quality, expert parallelism, prefix caching, and the people to operate it.

If you consume it as an API instead, a few things are worth knowing before you commit:

# K3 exposes an OpenAI-compatible interface. Via OpenRouter the model id is
# "moonshotai/kimi-k3"; Moonshot's direct API uses its own id and, unlike
# OpenRouter today, exposes prompt caching (cached input ~$0.30 / M tokens,
# a ~90% discount) which changes the economics of long, repeated contexts.
from openai import OpenAI

client = OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key="...",
)

resp = client.chat.completions.create(
    model="moonshotai/kimi-k3",          # 1,048,576-token context window
    messages=[{"role": "user", "content": "Summarise this repo's build failures."}],
    max_tokens=4096,                      # max output on the platform is 262,144
)
print(resp.choices[0].message.content)

Two operational caveats from independent testing that will not show up in a benchmark table. First, K3 reasons at maximum effort by default, and testers report it is slower and more verbose than the comparison median, which matters for latency-sensitive or cost-sensitive loops. Second, if you are in a regulated environment, Moonshot’s international privacy policy states personal data may be stored in Singapore, and its documents are not fully consistent on content use. Pricing snapshot for reference: OpenRouter listed K3 at $3 per million input tokens and $15 per million output tokens as of late July 2026. That is roughly 3x the input and 4x the output cost of the cheaper K2.7 Code release.

When the architecture actually matters to you

• If your workload is long-context and latency-or-cost-bound - agents that hold a large repository or document set in context and generate a lot - the KDA hybrid is the reason to care. The throughput and KV-cache claims target exactly your bottleneck. Pilot it and measure decode cost at your real context length.
• If you need an open-weight model you can host inside your own boundary, K3 is now the most capable option available, but budget for 8x H100 as a floor and staff for the operations. Do not assume it is cheaper than an API until you have modelled utilization.
• If your task is narrow, short-context coding at a price you already like, K3 is probably not your model. A smaller or cheaper option, including Moonshot’s own K2.7 Code, likely clears your bar for less.
• If you are quoting K3 to make a technology decision for someone else, separate the two claims cleanly: the architecture is real and independently interesting; the coding-benchmark supremacy is Moonshot-reported and unconfirmed.

The real question

The useful way to think about K3 is not “is it better than Fable 5.” It is “has hybrid linear attention crossed from research into a frontier-scale production model that people will actually deploy.” On that question the answer is yes, and that is the shift worth tracking - independent of whether Moonshot’s leaderboard claims survive scrutiny. The architecture is the contribution. The benchmarks are marketing until a third party reproduces them, and the open weights are a serious systems commitment before they are a cost saving.

If you want to evaluate it properly, ignore the 2.8T headline, read the Kimi Linear paper for the mechanism, and run your own long-context workload through it with the decode cost meter on. That is the only benchmark that will tell you anything about your system.

Sources

Papers and technical reports:

• Kimi Linear: An Expressive, Efficient Attention Architecture (arXiv:2510.26692) - the architecture K3 scales, and the primary source for KDA, the 3:1 hybrid, NoPE, and the throughput and KV-cache figures.
• Attention Residuals technical report (arXiv:2603.15031) - the AttnRes mechanism.

Vendor material:

• Kimi K3 tech blog, Moonshot AI
• Kimi K3 on OpenRouter - model id, context window, and pricing
• Kimi K3 day-0 support on vLLM - serving targets and hardware

Independent coverage and audits:

• Kimi K3 self-hosting, hardware, and benchmark write-up, Northflank
• Kimi K3 architecture breakdown - KDA, AttnRes, 896 experts, PoYo
• Kimi K3 benchmarks as a coding-agent evaluation guide, NxCode
• Kimi K3 technical FAQ - latent routing, quantile balancing, self-host math, RunPod
• Why K3’s expert parallelism is expensive to serve, F5
• What Kimi K3 is and why it is making waves, Bloomberg

All benchmark figures, pricing, and leaderboard positions are snapshots from late July 2026 and move quickly. Verify against the primary source before quoting any number from this post.