A Unified Approach to Controlling Implicit Regularization via Mirror Descent

TL;DR

This paper introduces a unified mirror descent framework to control implicit regularization in over-parameterized models, demonstrating its effectiveness in both regression and classification tasks with theoretical convergence guarantees.

Contribution

It presents a general mirror descent approach that unifies and extends implicit regularization control across various learning problems, addressing previous limitations.

Findings

MD converges to generalized maximum-margin solutions in classification.

MD can be efficiently implemented with fast convergence.

Different regularizers via MD lead to varied generalization performances.

Abstract

Inspired by the remarkable success of large neural networks, there has been significant interest in understanding the generalization performance of over-parameterized models. Substantial efforts have been invested in characterizing how optimization algorithms impact generalization through their "preferred" solutions, a phenomenon commonly referred to as implicit regularization. In particular, it has been argued that gradient descent (GD) induces an implicit $\ell_2$-norm regularization in regression and classification problems. However, the implicit regularization of different algorithms are confined to either a specific geometry or a particular class of learning problems, indicating a gap in a general approach for controlling the implicit regularization. To address this, we present a unified approach using mirror descent (MD), a notable generalization of GD, to control implicit regularization in both regression and classification settings. More specifically, we show that MD with the general class of homogeneous potential functions converges in direction to a generalized maximum-margin solution for linear classification problems, thereby answering a long-standing question in the classification setting. Further, we show that MD can be implemented efficiently and enjoys fast convergence under suitable conditions. Through comprehensive experiments, we demonstrate that MD is a versatile method to produce learned models with different regularizers, which in turn have different generalization performances.