Muon: The Optimizer That Ended AdamW's 12-Year Reign

For twelve years, almost every serious language model was trained the same way: with AdamW. That default is finally cracking. A new optimizer called Muon trains large models with roughly half the compute in some tests, and it may be quietly deciding which AI architectures look promising and which get thrown away.

Why the optimizer is the hard part

Training a model sounds simple: run it, measure the error, nudge the weights, repeat. Backpropagation gives you the gradient, which points toward the steepest increase in error, so you step the other way. What the gradient does not tell you is how big that step should be. Think of walking down a mountain at night: you can feel which way is downhill, but not whether to step an inch or ten meters.

The optimizer makes that call. Adam (adaptive moment estimation) became the standard because it handles two real problems. Training runs on small random batches, so gradients are noisy; Adam keeps a running average of past gradients to smooth them out. Different weights also behave very differently; Adam tracks a running average of squared gradients so each weight gets its own step size.

AdamW added one fix. Weight decay, a gentle pull that keeps weights from growing large and memorizing the training data, is applied separately from the gradient step instead of being mixed into it. That small change made training cleaner and more predictable, and AdamW became the default for over a decade.

What Muon actually does

Adam's blind spot is that it sees a model as one long list of numbers. But a transformer is built almost entirely from matrices: the attention projections, the feed-forward layers, the expert weights. A matrix is not a pile of independent values. It is a transformation that mixes inputs into new features, so its numbers are tied together by what they do as a group.

Picture 100 hikers heading down that dark mountain. Adam coaches each one individually. If 90 of them end up walking almost the same path, you have 100 people but only a handful of real routes being explored.

Any matrix update can be split into directions and strengths. Often a few directions carry nearly all the strength, so a big-looking update is really only moving the model along a few routes. Muon keeps the directions but evens out their strengths, so every direction contributes. That balancing step is the "orthogonalized" in its name, and it is what makes Muon matrix-aware.

The textbook way to do this is singular value decomposition (SVD), which breaks a matrix into clean perpendicular directions and their stretch factors. Running full SVD on every matrix at every step would cost far more than training itself. Muon instead uses Newton-Schulz iteration: apply a short matrix formula a few times and the update gets pushed close to orthogonal. It is not perfect, but it is far more balanced than where it started, and it uses only matrix multiplications, which is exactly what GPUs are best at.

The scaling problem, and how Kimi and DeepSeek beat it

Muon's strength is also its biggest practical headache. Adam scales easily because each weight updates on its own: split the model across thousands of GPUs and each one handles its slice. Muon needs the full matrix to orthogonalize it, but large matrices are routinely sharded across many GPUs. Using Muon means extra communication and waiting for the pieces to come together, on top of the communication costs distributed training already carries.

Kimi went first. Moonshot's paper "Muon is Scalable for LLM Training" was the first big evidence that Muon worked beyond small experiments. It added three fixes:

1. Weight decay. Plain Muon let some weights grow too large over long runs, risking late-training instability. AdamW-style decay kept them in check.
2. Update scaling. Muon's step size depends on a matrix's shape, so wide, tall and small matrices got wildly different update sizes. Per-parameter scaling made updates consistent across attention, MLP, expert and projection layers.
3. Distributed Muon. Built on a ZeRO-1 style setup: briefly gather the shards of a matrix, run Newton-Schulz, keep the local piece of the result and throw the rest away. Gather, compute, discard.

That is the setup behind the 52%-of-compute result.

DeepSeek V4 pushed further. It uses Muon for most modules, citing faster convergence and more stable training, but keeps AdamW for embeddings, prediction heads, static biases, gating factors and RMSNorm weights. Those are not ordinary 2D matrices, so element-wise optimization still fits them better.

DeepSeek also split Newton-Schulz into two phases. The first eight iterations use coefficients that drive singular values toward one quickly. The final two switch to stabilizing coefficients that hold them precisely at one without overshooting.

On the infrastructure side, DeepSeek capped how many GPUs can share any one dense matrix, so no matrix gets chopped into too many pieces. Gathering is still required, just across fewer GPUs. A knapsack-style assignment then balances which matrices land on which GPUs, so no single GPU gets stuck with all the big ones and stalls the step.

If two frontier labs have made Muon work at scale and published how, others can follow. Even a fraction of Kimi's gain, say 10% of compute, is a lot of money at frontier scale.

The bigger story: the optimizer was judging the architectures

For a decade, AI research ran on a quiet assumption: test new ideas with AdamW. If an architecture trained well, it was promising. If not, it was probably a dead end. Muon raises an uncomfortable question. What if some of those ideas failed because of the optimizer, not the idea?

Parallax is the first clear example. Tilde Research revived local linear attention. Standard softmax attention effectively averages the values of nearby keys, treating the neighborhood around a query as flat. Local linear attention fits a small linear trend there instead, which should be more expressive. The original version was expensive, numerically unstable and awkward to pre-train, so it went nowhere. Parallax sidesteps the costly solve by learning an extra query-like projection that approximates the correction.

The twist: Parallax's edge depends heavily on Muon. Under AdamW, it and softmax attention look roughly even. Under Muon, Parallax pulls clearly ahead. Same architecture, different verdict, decided by the optimizer.

Why would Muon unlock ideas? Another recent paper argues Muon does not just train faster, it learns better features:

• More robust. Muon-trained models hold up better when images or text are corrupted with noise, compared with Adam or plain SGD.
• More transferable. Frozen Muon-pretrained backbones, or fine-tuned ones, adapt better to new downstream tasks.

The paper ties this to two mechanisms. Muon models show larger logit margins, meaning the right answer beats the strongest wrong answer by more, so noise has to push harder to flip a prediction. They also show higher effective rank: hidden states use more independent directions instead of collapsing into a few dominant ones. That is exactly what you would expect from an optimizer that balances every direction in each update.

What comes next

Muon's headline is cheaper training, and that alone would make it a big deal. The deeper shift is methodological. If the choice of optimizer can flip an architecture from "meh" to "clearly better," then a lot of shelved ideas deserve a rerun, and every new architecture paper should probably report results under more than one optimizer.

And there is no reason to think Muon is the end of the line. It is simply the first method in twelve years to seriously dent AdamW's dominance. Once one assumption falls, the next ones get easier to question.

Lyle Heartman writes about the ideas shaping modern AI.