Entropic Mirror Descent for Linear Systems: Polyak's Stepsize and Implicit Bias

TL;DR

This paper develops an entropic mirror descent method with Polyak's stepsize for linear systems, providing convergence guarantees and implicit bias analysis without restrictive assumptions.

Contribution

It introduces a novel variant of Polyak stepsizes for entropic mirror descent, strengthening implicit bias bounds and extending convergence results to convex smooth functions.

Findings

Achieves sublinear and linear convergence rates.

Strengthens bounds on $\, ext{l}_1$-norm implicit bias.

Proposes an exponentiation-free alternative method.

Abstract

This paper focuses on applying entropic mirror descent to solve linear systems, where the main challenge for the convergence analysis stems from the unboundedness of the domain. To overcome this without imposing restrictive assumptions, we introduce a variant of Polyak-type stepsizes. Along the way, we strengthen the bound for $\ell_1$-norm implicit bias, obtain sublinear and linear convergence results, and generalize the convergence result to arbitrary convex $L$-smooth functions. We also propose an alternative method that avoids exponentiation, resembling the original Hadamard descent, but with provable convergence.