Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation

TL;DR

Pion is a novel spectrum-preserving optimizer for large language models that uses orthogonal transformations to maintain weight matrix singular values, offering stability and competitive performance.

Contribution

We propose Pion, a new optimizer based on orthogonal equivalence transformation that preserves spectral properties of weights during LLM training.

Findings

Pion maintains spectral norms throughout training.

Empirical results show Pion is stable and competitive.

Pion effectively modulates weight matrix geometry.

Abstract

We introduce Pion, a spectrum-preserving optimizer for large language model (LLM) training based on orthogonal equivalence transformation. Unlike additive optimizers such as Adam and Muon, Pion updates each weight matrix through left and right orthogonal transformations, preserving its singular values throughout training. This yields an optimization mechanism that modulates the geometry of weight matrices while keeping their spectral norm fixed. We derive the Pion update rule, systematically examine its design choices, and analyze its convergence behavior along with several key properties. Empirical results show that Pion offers a stable and competitive alternative to standard optimizers for both LLM pretraining and finetuning.