Stabilizing Native Low-Rank LLM Pretraining

TL;DR

This paper introduces Spectron, a spectral normalization technique that stabilizes low-rank training of large language models from scratch, enabling efficient, stable, and high-performance factorized training without auxiliary guidance.

Contribution

The paper presents a novel spectral renormalization method, Spectron, for stable native low-rank training of LLMs, and establishes compute-optimal scaling laws for such models.

Findings

Stable low-rank training achieved without auxiliary guidance.

Spectron effectively bounds spectral norm growth during training.

Low-rank models demonstrate predictable power-law scaling and improved efficiency.

Abstract

Foundation models have achieved remarkable success, yet their growing parameter counts pose significant computational and memory challenges. Low-rank factorization offers a promising route to reduce training and inference costs, but the community lacks a stable recipe for training models from scratch using exclusively low-rank weights while matching the performance of the dense model. We demonstrate that Large Language Models (LLMs) can be trained from scratch using exclusively low-rank factorized weights for all non-embedding matrices without auxiliary "full-rank" guidance required by prior methods. While native low-rank training often suffers from instability and loss spikes, we identify uncontrolled growth in the spectral norm (largest singular value) of the weight matrix update as the dominant factor. To address this, we introduce Spectron: Spectral renormalization with orthogonalization, which dynamically bounds the resultant weight updates based on the current spectral norms of the factors. Our method enables stable, end-to-end factorized training with negligible overhead. Finally, we establish compute-optimal scaling laws for natively low-rank transformers, demonstrating predictable power-law behavior and improved inference efficiency relative to dense models.