Magnificent Minified Models

TL;DR

This paper evaluates various neural network compression techniques, including parameter pruning, neuron pruning, and quantization, comparing their effectiveness and impact on model accuracy, and examines their relation to the Lottery Ticket Hypothesis.

Contribution

It introduces OBD-SD, a variation of Optimal Brain Damage, and provides a comprehensive comparison of pruning and quantization methods, highlighting their relative performance and insights into the Lottery Ticket Hypothesis.

Findings

OBD-SD slightly outperforms other pruning methods.

Retraining from scratch is more effective for neuron pruning.

Fine-tuning pruned models marginally better than retraining from scratch.

Abstract

This paper concerns itself with the task of taking a large trained neural network and 'compressing' it to be smaller by deleting parameters or entire neurons, with minimal decreases in the resulting model accuracy. We compare various methods of parameter and neuron selection: dropout-based neuron damage estimation, neuron merging, absolute-value based selection, random selection, OBD (Optimal Brain Damage). We also compare a variation on the classic OBD method that slightly outperformed all other parameter and neuron selection methods in our tests with substantial pruning, which we call OBD-SD. We compare these methods against quantization of parameters. We also compare these techniques (all applied to a trained neural network), with neural networks trained from scratch (random weight initialization) on various pruned architectures. Our results are only barely consistent with the Lottery Ticket Hypothesis, in that fine-tuning a parameter-pruned model does slightly better than retraining a similarly pruned model from scratch with randomly initialized weights. For neuron-level pruning, retraining from scratch did much better in our experiments.