The Effect of Data Dimensionality on Neural Network Prunability

TL;DR

This paper investigates how the intrinsic low-dimensional structure of high-dimensional input data influences the extent to which neural networks can be pruned without losing accuracy, highlighting a factor affecting model efficiency.

Contribution

It is the first study to analyze the impact of input data's low-dimensional manifold structure on neural network prunability.

Findings

Low-dimensional input structures can increase neural network prunability.

Prunability varies significantly with the intrinsic data dimensionality.

Understanding data structure helps optimize pruning strategies.

Abstract

Practitioners prune neural networks for efficiency gains and generalization improvements, but few scrutinize the factors determining the prunability of a neural network the maximum fraction of weights that pruning can remove without compromising the model's test accuracy. In this work, we study the properties of input data that may contribute to the prunability of a neural network. For high dimensional input data such as images, text, and audio, the manifold hypothesis suggests that these high dimensional inputs approximately lie on or near a significantly lower dimensional manifold. Prior work demonstrates that the underlying low dimensional structure of the input data may affect the sample efficiency of learning. In this paper, we investigate whether the low dimensional structure of the input data affects the prunability of a neural network.