An Adaptive Tangent Feature Perspective of Neural Networks

TL;DR

This paper introduces an adaptive tangent feature framework for neural networks, providing new insights into feature learning, kernel alignment, and low-rank solutions through a joint optimization approach.

Contribution

It proposes a novel framework for understanding neural network feature learning via tangent features with feature transformations, linking to structured regularization and kernel alignment.

Findings

Features and kernel functions evolve during training.

Structured regularization encourages low-rank solutions.

Kernel alignment improves with adaptive tangent features.

Abstract

In order to better understand feature learning in neural networks, we propose a framework for understanding linear models in tangent feature space where the features are allowed to be transformed during training. We consider linear transformations of features, resulting in a joint optimization over parameters and transformations with a bilinear interpolation constraint. We show that this optimization problem has an equivalent linearly constrained optimization with structured regularization that encourages approximately low rank solutions. Specializing to neural network structure, we gain insights into how the features and thus the kernel function change, providing additional nuance to the phenomenon of kernel alignment when the target function is poorly represented using tangent features. We verify our theoretical observations in the kernel alignment of real neural networks.