When Layers Play the Lottery, all Tickets Win at Initialization

TL;DR

This paper explores layer pruning in neural networks, demonstrating that sparse subnetworks, or winning tickets, can be identified at initialization without training, leading to faster, greener, and more robust AI models.

Contribution

It introduces a novel layer pruning approach to find winning tickets at initialization, reducing training costs and improving robustness compared to traditional filter pruning methods.

Findings

Winning tickets exist when layers are pruned.

Layer-based winning tickets speed up training and reduce carbon emissions.

Subnetworks from layer pruning are more robust against adversarial and out-of-distribution data.

Abstract

Pruning is a standard technique for reducing the computational cost of deep networks. Many advances in pruning leverage concepts from the Lottery Ticket Hypothesis (LTH). LTH reveals that inside a trained dense network exists sparse subnetworks (tickets) able to achieve similar accuracy (i.e., win the lottery - winning tickets). Pruning at initialization focuses on finding winning tickets without training a dense network. Studies on these concepts share the trend that subnetworks come from weight or filter pruning. In this work, we investigate LTH and pruning at initialization from the lens of layer pruning. First, we confirm the existence of winning tickets when the pruning process removes layers. Leveraged by this observation, we propose to discover these winning tickets at initialization, eliminating the requirement of heavy computational resources for training the initial (over-parameterized) dense network. Extensive experiments show that our winning tickets notably speed up the training phase and reduce up to 51% of carbon emission, an important step towards democratization and green Artificial Intelligence. Beyond computational benefits, our winning tickets exhibit robustness against adversarial and out-of-distribution examples. Finally, we show that our subnetworks easily win the lottery at initialization while tickets from filter removal (the standard structured LTH) hardly become winning tickets.