FlipOut: Uncovering Redundant Weights via Sign Flipping

TL;DR

FlipOut is a novel neural network pruning method that leverages sign flips during training to identify redundant weights, enabling effective pruning before convergence with minimal tuning and achieving state-of-the-art sparsity levels.

Contribution

This paper introduces FlipOut, a pruning technique that uses weight sign oscillations to determine saliency, allowing early pruning with minimal hyperparameter tuning.

Findings

Achieves state-of-the-art sparsity levels of 99.6% and above.

Performs pruning before network convergence.

Requires minimal hyperparameter tuning.

Abstract

Modern neural networks, although achieving state-of-the-art results on many tasks, tend to have a large number of parameters, which increases training time and resource usage. This problem can be alleviated by pruning. Existing methods, however, often require extensive parameter tuning or multiple cycles of pruning and retraining to convergence in order to obtain a favorable accuracy-sparsity trade-off. To address these issues, we propose a novel pruning method which uses the oscillations around $0$ (i.e. sign flips) that a weight has undergone during training in order to determine its saliency. Our method can perform pruning before the network has converged, requires little tuning effort due to having good default values for its hyperparameters, and can directly target the level of sparsity desired by the user. Our experiments, performed on a variety of object classification architectures, show that it is competitive with existing methods and achieves state-of-the-art performance for levels of sparsity of $99.6\%$ and above for most of the architectures tested. For reproducibility, we release our code publicly at https://github.com/AndreiXYZ/flipout.