FISMO: Fisher-Structured Momentum-Orthogonalized Optimizer

TL;DR

FISMO introduces a Fisher-geometry-based optimizer that balances isotropic and anisotropic updates, improving training efficiency and performance in large-scale neural network optimization.

Contribution

It generalizes momentum-orthogonalized updates to incorporate Fisher information geometry, providing convergence guarantees and better adaptation to loss landscape curvature.

Findings

FISMO outperforms baseline optimizers in training efficiency.

FISMO achieves higher final accuracy on benchmarks.

Theoretical convergence rate of O(1/√T) established.

Abstract

Training large-scale neural networks requires solving nonconvex optimization where the choice of optimizer fundamentally determines both convergence behavior and computational efficiency. While adaptive methods like Adam have long dominated practice, the recently proposed Muon optimizer achieves superior performance through orthogonalized momentum updates that enforce isotropic geometry with uniform singular values. However, this strict isotropy discards potentially valuable curvature information encoded in gradient spectra, motivating optimization methods that balance geometric structure with adaptivity. We introduce FISMO (Fisher-Structured Momentum-Orthogonalized) optimizer, which generalizes isotropic updates to incorporate anisotropic curvature information through Fisher information geometry. By reformulating the optimizer update as a trust-region problem constrained by a Kronecker-factored Fisher metric, FISMO achieves structured preconditioning that adapts to local loss landscape geometry while maintaining computational tractability. We establish convergence guarantees for FISMO in stochastic nonconvex settings, proving an $\mathcal{O}(1/\sqrt{T})$ rate for the expected squared gradient norm with explicit characterization of variance reduction through mini-batching. Empirical evaluation on image classification and language modeling benchmarks demonstrates that FISMO achieves superior training efficiency and final performance compared to established baselines.