Adaptive Matrix Online Learning through Smoothing with Guarantees for Nonsmooth Nonconvex Optimization

TL;DR

This paper introduces adaptive matrix online learning algorithms that use smoothing techniques to achieve regret bounds comparable to existing methods but with reduced computational complexity, and extends guarantees to nonsmooth nonconvex optimization.

Contribution

It develops efficient adaptive algorithms for matrix online learning using smoothing, providing regret guarantees and convergence analysis in nonsmooth nonconvex settings.

Findings

Achieves regret bounds matching one-sided Shampoo with less computation.

Introduces two smoothing-based methods: FTPL with Gaussian smoothing and FAML with hyperbolic smoothing.

Provides convergence guarantees for the derived matrix optimizers in nonsmooth nonconvex optimization.

Abstract

We study online linear optimization with matrix variables constrained by the operator norm, a setting where the geometry renders designing data-dependent and efficient adaptive algorithms challenging. The best-known adaptive regret bounds are achieved by Shampoo-like methods, but they require solving a costly quadratic projection subproblem. To address this, we extend the gradient-based prediction scheme to adaptive matrix online learning and cast algorithm design as constructing a family of smoothed potentials for the nuclear norm. We define a notion of admissibility for such smoothings and prove any admissible smoothing yields a regret bound matching the best-known guarantees of one-sided Shampoo. We instantiate this framework with two efficient methods that avoid quadratic projections. The first is an adaptive Follow-the-Perturbed-Leader (FTPL) method using Gaussian stochastic smoothing. The second is Follow-the-Augmented-Matrix-Leader (FAML), which uses a deterministic hyperbolic smoothing in an augmented matrix space. By analyzing the admissibility of these smoothings, we show both methods admit closed-form updates and match one-sided Shampoo's regret up to a constant factor, while significantly reducing computational cost. Lastly, using the online-to-nonconvex conversion, we derive two matrix-based optimizers, Pion (from FTPL) and Leon (from FAML). We prove convergence guarantees for these methods in nonsmooth nonconvex settings, a guarantee that the popular Muon optimizer lacks.