# The Intrinsic Dimension of Neural Network Ensembles

## TL;DR

This paper explores how different training strategies affect the variability and performance of neural network ensembles.

## Contribution

The study introduces intrinsic dimension as a novel measure to quantify variability in neural network ensembles.

## Key findings

- Random initialization causes more variability than data distortion, dropout, or batch shuffle.
- Intrinsic dimension reflects the impact of training strategies on parameter space coverage.
- Training choices significantly affect prediction accuracy and ensemble diversity.

## Abstract

In this work, we propose to study the collective behavior of different ensembles of neural networks. These sets define and live on complex manifolds that evolve through training. Each manifold is characterized by its intrinsic dimension, a measure of the variability of the ensemble and, as such, a measure of the impact of the different training strategies. Indeed, higher intrinsic dimension values imply higher variability among the networks and a larger parameter space coverage. Here, we quantify how much the training choices allow the exploration of the parameter space, finding that a random initialization of the parameters is a stronger source of variability than, progressively, data distortion, dropout, and batch shuffle. We then investigate the combinations of these strategies, the parameters involved, and the impact on the accuracy of the predictions, shedding light on the often-underestimated consequences of these training choices.