OLion: Approaching the Hadamard Ideal by Intersecting Spectral and $\ell_{\infty}$ Implicit Biases

TL;DR

OLion introduces a novel optimizer that combines spectral and coordinate control to efficiently approximate the Hadamard ideal, demonstrating superior performance in large-scale language and vision training tasks.

Contribution

The paper presents OLion, a new optimizer that integrates spectral control with $ ext{l}_ ext{infty}$ biases, providing an efficient approximation to the Hadamard ideal with convergence guarantees.

Findings

Matches or outperforms AdamW and Muon in large-scale training

Mitigates optimizer mismatch during fine-tuning

Proven convergence under mild assumptions

Abstract

Many optimizers can be interpreted as steepest-descent methods under norm-induced geometries, and thus inherit corresponding implicit biases. We introduce \nameA{} (\fullname{}), which combines spectral control from orthogonalized update directions with $\ell_\infty$-style coordinate control from sign updates. \nameA{} forms a Lion-style momentum direction, approximately orthogonalizes it via a few Newton--Schulz iterations, and then applies an entrywise sign, providing an efficient approximation to taking a maximal step over the intersection of the spectral and $\ell_\infty$ constraint sets (a scaled Hadamard-like set for matrix parameters). Despite the strong nonlinearity of orthogonalization and sign, we prove convergence under a mild, empirically verified diagonal-isotropy assumption. Across large-scale language and vision training, including GPT-2 and Llama pretraining, SiT image pretraining, and supervised fine-tuning, \nameA{} matches or outperforms AdamW and Muon under comparable tuning while using only momentum-level optimizer state, and it mitigates optimizer mismatch when fine-tuning AdamW-pretrained checkpoints.