Sparsity-aware generalization theory for deep neural networks

TL;DR

This paper introduces a new theoretical framework for understanding how sparsity in hidden layer activations influences the generalization ability of deep neural networks, providing bounds that improve upon existing norm-based methods.

Contribution

It develops a sparsity-aware generalization theory that accounts for effective model size reduction due to activation sparsity without strong assumptions.

Findings

Shows fundamental trade-offs between sparsity and generalization.

Provides non-vacuous bounds with data-dependent priors in over-parametrized models.

Improves over recent norm-based approaches.

Abstract

Deep artificial neural networks achieve surprising generalization abilities that remain poorly understood. In this paper, we present a new approach to analyzing generalization for deep feed-forward ReLU networks that takes advantage of the degree of sparsity that is achieved in the hidden layer activations. By developing a framework that accounts for this reduced effective model size for each input sample, we are able to show fundamental trade-offs between sparsity and generalization. Importantly, our results make no strong assumptions about the degree of sparsity achieved by the model, and it improves over recent norm-based approaches. We illustrate our results numerically, demonstrating non-vacuous bounds when coupled with data-dependent priors in specific settings, even in over-parametrized models.