Deconstructing the Structure of Sparse Neural Networks

TL;DR

This paper analyzes the structure of sparse neural networks, revealing that accuracy can depend solely on structure, and proposes more efficient training methods based on early structure emergence.

Contribution

It introduces a structural perspective to sparse neural networks, demonstrating structure's role in accuracy and proposing an early-structure detection algorithm.

Findings

Accuracy can be derived from network structure alone

Structural robustness varies across algorithms

Early structure emergence enables more efficient training

Abstract

Although sparse neural networks have been studied extensively, the focus has been primarily on accuracy. In this work, we focus instead on network structure, and analyze three popular algorithms. We first measure performance when structure persists and weights are reset to a different random initialization, thereby extending experiments in Deconstructing Lottery Tickets (Zhou et al., 2019). This experiment reveals that accuracy can be derived from structure alone. Second, to measure structural robustness we investigate the sensitivity of sparse neural networks to further pruning after training, finding a stark contrast between algorithms. Finally, for a recent dynamic sparsity algorithm we investigate how early in training the structure emerges. We find that even after one epoch the structure is mostly determined, allowing us to propose a more efficient algorithm which does not require dense gradients throughout training. In looking back at algorithms for sparse neural networks and analyzing their performance from a different lens, we uncover several interesting properties and promising directions for future research.