Implicit Regularization via Neural Feature Alignment

TL;DR

This paper investigates how neural networks implicitly regularize during training by aligning their features along task-relevant directions, leading to feature selection and compression, supported by a new complexity measure based on tangent kernel analysis.

Contribution

It introduces a geometric perspective on implicit regularization, highlighting feature alignment effects and proposing a heuristic complexity measure related to tangent kernel evolution.

Findings

Neural tangent features align along task-relevant directions during training.

Feature alignment acts as a form of implicit regularization.

A new complexity measure based on tangent kernel sequences explains this phenomenon.

Abstract

We approach the problem of implicit regularization in deep learning from a geometrical viewpoint. We highlight a regularization effect induced by a dynamical alignment of the neural tangent features introduced by Jacot et al, along a small number of task-relevant directions. This can be interpreted as a combined mechanism of feature selection and compression. By extrapolating a new analysis of Rademacher complexity bounds for linear models, we motivate and study a heuristic complexity measure that captures this phenomenon, in terms of sequences of tangent kernel classes along optimization paths.