Supporting Evidence for the Adaptive Feature Program across Diverse Models

TL;DR

This paper provides theoretical support for the adaptive feature program in neural networks by analyzing feature learning dynamics across various models, suggesting its potential effectiveness.

Contribution

It introduces the feature error measure (FEM) and demonstrates its decrease during training in multiple adaptive feature models, supporting the adaptive feature program.

Findings

FEM decreases during training in linear regression models.

FEM decreases in single and multiple index models.

Supports the potential success of the adaptive feature program.

Abstract

Theoretically exploring the advantages of neural networks might be one of the most challenging problems in the AI era. An adaptive feature program has recently been proposed to analyze feature learning, the characteristic property of neural networks, in a more abstract way. Motivated by the celebrated Le Cam equivalence, we advocate the over-parameterized sequence models to further simplify the analysis of the training dynamics of adaptive feature program and present several pieces of supporting evidence for the adaptive feature program. More precisely, after having introduced the feature error measure (FEM) to characterize the quality of the learned feature, we show that the FEM is decreasing during the training process of several concrete adaptive feature models including linear regression, single/multiple index models, etc. We believe that this hints at the potential successes of the adaptive feature program.