IFNSO: Iteration-Free Newton-Schulz Orthogonalization

TL;DR

IFNSO introduces an iteration-free approach to Newton-Schulz orthogonalization, significantly reducing computational costs while maintaining stable convergence, thus improving efficiency in orthogonalization tasks for optimization.

Contribution

The paper presents a novel iteration-free framework that simplifies Newton-Schulz orthogonalization by using a learnable polynomial, enhancing efficiency and stability.

Findings

IFNSO outperforms existing methods in efficiency.

The approach maintains stable convergence.

Experimental results validate superior performance.

Abstract

The Newton-Schulz (NS) iteration has become a key technique for orthogonalization in optimizers such as Muon and for optimization on the Stiefel manifold. Despite its effectiveness, the conventional NS iteration incurs significant computational overhead due to repeated high-dimensional matrix multiplications. To overcome these limitations, we propose Iteration-Free Newton-Schulz Orthogonalization (IFNSO), a novel framework that consolidates the traditional iterative structure into a unified and Iteration-Free formulation. By analyzing the contribution of individual matrix powers, we streamline the process by removing insignificant terms and introducing a polynomial with learnable coefficients. These coefficients are optimized to ensure both superior computational efficiency and stable convergence. Extensive experiments demonstrate that IFNSO achieves superior performance compared to existing methods. Our code is available at: https://github.com/greekinRoma/Unified_Newton_Schulz_Orthogonalization.