Learning Sparse Networks Using Targeted Dropout

TL;DR

This paper introduces targeted dropout, a simple training method that makes neural networks more robust to pruning by selectively dropping units or weights during training, leading to more efficient sparse networks.

Contribution

The paper proposes targeted dropout, a novel stochastic dropout technique that enhances neural network pruning efficiency and robustness compared to existing regularizers.

Findings

Improves network sparsity with minimal performance loss

Simpler to implement than existing regularizers

Achieves better pruning robustness in experiments

Abstract

Neural networks are easier to optimise when they have many more weights than are required for modelling the mapping from inputs to outputs. This suggests a two-stage learning procedure that first learns a large net and then prunes away connections or hidden units. But standard training does not necessarily encourage nets to be amenable to pruning. We introduce targeted dropout, a method for training a neural network so that it is robust to subsequent pruning. Before computing the gradients for each weight update, targeted dropout stochastically selects a set of units or weights to be dropped using a simple self-reinforcing sparsity criterion and then computes the gradients for the remaining weights. The resulting network is robust to post hoc pruning of weights or units that frequently occur in the dropped sets. The method improves upon more complicated sparsifying regularisers while being simple to implement and easy to tune.