Considering Layerwise Importance in the Lottery Ticket Hypothesis

TL;DR

This paper extends the Lottery Ticket Hypothesis by incorporating layerwise importance metrics, revealing multiple distinct sparse subnetworks and challenging the notion of a unique lottery ticket.

Contribution

It introduces a method to consider weight importance within layers, demonstrating the non-uniqueness of lottery tickets across different importance metrics.

Findings

Different importance metrics produce distinct lottery tickets.

Lottery tickets are not unique, with little overlap in selected connections.

Layerwise importance enhances understanding of sparse network extraction.

Abstract

The Lottery Ticket Hypothesis (LTH) showed that by iteratively training a model, removing connections with the lowest global weight magnitude and rewinding the remaining connections, sparse networks can be extracted. This global comparison removes context information between connections within a layer. Here we study means for recovering some of this layer distributional context and generalise the LTH to consider weight importance values rather than global weight magnitudes. We find that given a repeatable training procedure, applying different importance metrics leads to distinct performant lottery tickets with little overlapping connections. This strongly suggests that lottery tickets are not unique