# Information Losses in Neural Classifiers from Sampling

## TL;DR

This paper investigates how finite training datasets cause information loss in neural classifiers, providing bounds that are less sensitive to input compression and align well with experimental observations.

## Contribution

It establishes a relationship between information loss and total variation of neural models, deriving dataset size bounds that improve upon previous bounds without relying on model complexity.

## Key findings

- Bounds on information loss are smaller and less sensitive to input compression.
- The bounds align well with experimental results on neural network information compression.
- Theoretical insights explain recent experimental observations of information compression.

## Abstract

This paper considers the subject of information losses arising from the finite datasets used in the training of neural classifiers. It proves a relationship between such losses as the product of the expected total variation of the estimated neural model with the information about the feature space contained in the hidden representation of that model. It then bounds this expected total variation as a function of the size of randomly sampled datasets in a fairly general setting, and without bringing in any additional dependence on model complexity. It ultimately obtains bounds on information losses that are less sensitive to input compression and in general much smaller than existing bounds. The paper then uses these bounds to explain some recent experimental findings of information compression in neural networks which cannot be explained by previous work. Finally, the paper shows that not only are these bounds much smaller than existing ones, but that they also correspond well with experiments.