Convolutional and Residual Networks Provably Contain Lottery Tickets

TL;DR

This paper proves that modern neural network architectures, including convolutional and residual networks with various activation functions, likely contain sparse subnetworks or 'lottery tickets' that can perform well after pruning.

Contribution

It extends the theoretical understanding of lottery tickets to convolutional and residual architectures, beyond fully-connected networks.

Findings

Convolutional and residual networks contain lottery tickets with high probability.

Theoretical proof applies to networks with diverse activation functions.

Supports practical pruning strategies for modern neural networks.

Abstract

The Lottery Ticket Hypothesis continues to have a profound practical impact on the quest for small scale deep neural networks that solve modern deep learning tasks at competitive performance. These lottery tickets are identified by pruning large randomly initialized neural networks with architectures that are as diverse as their applications. Yet, theoretical insights that attest their existence have been mostly focused on deep fully-connected feed forward networks with ReLU activation functions. We prove that also modern architectures consisting of convolutional and residual layers that can be equipped with almost arbitrary activation functions can contain lottery tickets with high probability.