Learning distinct features helps, provably

TL;DR

This paper provides a theoretical analysis showing that learning diverse, non-redundant features in neural networks improves their generalization ability, supported by novel bounds based on feature diversity.

Contribution

The paper introduces a theoretical framework linking feature diversity to generalization in neural networks, with bounds derived for two-layer and extended to deeper architectures.

Findings

More distinct features lead to better generalization.

Feature diversity can be quantified and bounded using Rademacher complexity.

Results extend to deeper networks and various loss functions.

Abstract

We study the diversity of the features learned by a two-layer neural network trained with the least squares loss. We measure the diversity by the average $L_2$-distance between the hidden-layer features and theoretically investigate how learning non-redundant distinct features affects the performance of the network. To do so, we derive novel generalization bounds depending on feature diversity based on Rademacher complexity for such networks. Our analysis proves that more distinct features at the network's units within the hidden layer lead to better generalization. We also show how to extend our results to deeper networks and different losses.