Implicit Bias of Mirror Flow on Separable Data

TL;DR

This paper investigates how mirror flow, a continuous-time version of mirror descent, converges to maximum margin classifiers on linearly separable data, influenced by the choice of mirror potential.

Contribution

It characterizes the implicit bias of mirror flow towards a maximum margin classifier determined by the horizon function of the mirror potential.

Findings

Mirror flow converges in direction to a $\,\,\, ext{phi}_ ext{infty}$-maximum margin classifier.

The shape of the horizon function at infinity influences the solution bias.

Numerical experiments confirm theoretical predictions across different potentials.

Abstract

We examine the continuous-time counterpart of mirror descent, namely mirror flow, on classification problems which are linearly separable. Such problems are minimised `at infinity' and have many possible solutions; we study which solution is preferred by the algorithm depending on the mirror potential. For exponential tailed losses and under mild assumptions on the potential, we show that the iterates converge in direction towards a $\phi_\infty$-maximum margin classifier. The function $\phi_\infty$ is the \textit{horizon function} of the mirror potential and characterises its shape `at infinity'. When the potential is separable, a simple formula allows to compute this function. We analyse several examples of potentials and provide numerical experiments highlighting our results.