On the connections between algorithmic regularization and penalization for convex losses

TL;DR

This paper demonstrates that under certain geometric conditions, the iterative optimization paths for convex loss functions can be equivalently represented as solutions to regularized penalized problems, linking algorithmic regularization and explicit penalization.

Contribution

It introduces a geometric condition that characterizes when the optimization path of an iterative algorithm matches a penalized problem's solution path for convex losses.

Findings

Establishes the equivalence between algorithmic regularization and penalization for convex losses.

Provides a geometric condition for the optimization path of convex functions.

Shows the representation of iterative algorithm paths as penalized solutions.

Abstract

In this work we establish the equivalence of algorithmic regularization and explicit convex penalization for generic convex losses. We introduce a geometric condition for the optimization path of a convex function, and show that if such a condition is satisfied, the optimization path of an iterative algorithm on the unregularized optimization problem can be represented as the solution path of a corresponding penalized problem.