Iterative Orthogonalization Scaling Laws

TL;DR

This paper investigates the scaling behavior of the iterative orthogonalization process in the muon optimizer, revealing potential issues at large scales due to shrinking singular values, supported by theoretical and empirical analysis.

Contribution

It provides a theoretical and empirical analysis of the scaling laws affecting the orthogonalization process in the muon optimizer at large scales.

Findings

Singular values of random matrices shrink with scale.

The orthogonalization procedure may face issues at large scales.

The paper does not propose solutions to the identified problem.

Abstract

The muon optimizer has picked up much attention as of late as a possible replacement to the seemingly omnipresent Adam optimizer. Recently, care has been taken to document the scaling laws of hyper-parameters under muon such as weight decay and learning rate. However, at much larger scales the iterative orthogonalization procedure present in muon may suffer a possible issue as the singular values of random matrices shrink with scale. This paper shows this scaling behavior theoretically and empirically on random matrices but does not suggest what to do about it.