Data-dependent Pruning to find the Winning Lottery Ticket

TL;DR

This paper investigates data-dependent pruning methods for neural networks, demonstrating that incorporating gradient information into the pruning criterion enhances the ability to find subnetworks that match full network performance, supporting the Lottery Ticket Hypothesis.

Contribution

It introduces a data-dependent pruning approach using gradient information, improving subnetwork identification compared to previous methods.

Findings

Data-dependent pruning improves subnetwork performance

Gradient-based criteria outperform traditional pruning methods

Supports the Lottery Ticket Hypothesis with enhanced pruning techniques

Abstract

The Lottery Ticket Hypothesis postulates that a freshly initialized neural network contains a small subnetwork that can be trained in isolation to achieve similar performance as the full network. Our paper examines several alternatives to search for such subnetworks. We conclude that incorporating a data dependent component into the pruning criterion in the form of the gradient of the training loss -- as done in the SNIP method -- consistently improves the performance of existing pruning algorithms.