Implicit Bias of Mirror Flow in Homogeneous Neural Networks: Sparse and Dense Feature Learning

TL;DR

This paper analyzes how mirror flow influences max-margin solutions in homogeneous neural networks, revealing diverse feature learning behaviors and convergence properties through theoretical and experimental insights.

Contribution

It extends classical gradient flow results to mirror flow, deriving a novel balance equation and characterizing max-margin solutions with convergence and norm estimates.

Findings

Different mirror maps can lead to the same max-margin solution.

Convergence can be exponentially slow.

Mirror maps can produce sparse or dense neuron activations.

Abstract

We study the max-margin solutions reached by mirror flow in deep neural networks with homogeneous activation functions. Extending classical results on gradient flow, we derive a novel balance equation for mirror flow from convex duality, enabling a characterization of the horizon function governing the induced margin. We further establish max-margin characterizations together with convergence rates and norm growth estimates. Finally, we support our theory through experiments on synthetic datasets and standard vision tasks. Concretely, we show that: (1) distinct non-homogeneous mirror maps can induce the same max-margin solution; (2) convergence can be extremely slow, including exponentially slow regimes; and (3) although all considered mirror maps exhibit feature learning, they can produce markedly different representations, ranging from sparse to dense neuron activations. Together, these results provide a unified perspective on sparse and dense feature learning in homogeneous neural networks, highlighting how mirror maps shape both optimization dynamics and the geometry of the learned classifiers.