Muon is a strong optimizer for matrix-valued parameters in large language model pretraining that approximately orthogonalizes momentum using Newton-Schulz iterations

Muon has emerged as a strong optimizer for the matrix-valued parameters in large language model pretraining, approximately orthogonalizing its momentum with a few Newton-Schulz iterations.

http://arxiv.org/abs/2608.26288v1