Implicit Bias of Mirror Flow for Shallow Neural Networks in Univariate Regression

TL;DR

This paper investigates the implicit bias of mirror flow in shallow neural networks for univariate regression, revealing how different potentials influence training behavior and bias, including lazy training and kernel regimes.

Contribution

It characterizes the implicit bias of mirror flow for wide shallow networks, extending prior results and analyzing effects of scaled potentials and activation functions.

Findings

Mirror flow exhibits lazy training similar to gradient flow in wide networks.

Scaled potentials can induce biases outside the kernel regime.

Biases depend on the potential and activation, affecting curvature penalization.

Abstract

We examine the implicit bias of mirror flow in univariate least squares error regression with wide and shallow neural networks. For a broad class of potential functions, we show that mirror flow exhibits lazy training and has the same implicit bias as ordinary gradient flow when the network width tends to infinity. For ReLU networks, we characterize this bias through a variational problem in function space. Our analysis includes prior results for ordinary gradient flow as a special case and lifts limitations which required either an intractable adjustment of the training data or networks with skip connections. We further introduce scaled potentials and show that for these, mirror flow still exhibits lazy training but is not in the kernel regime. For networks with absolute value activations, we show that mirror flow with scaled potentials induces a rich class of biases, which generally cannot be captured by an RKHS norm. A takeaway is that whereas the parameter initialization determines how strongly the curvature of the learned function is penalized at different locations of the input space, the scaled potential determines how the different magnitudes of the curvature are penalized.