Neural Network Topologies for Sparse Training

TL;DR

This paper introduces a deterministic algorithm to generate diverse sparse neural network topologies that maintain the training efficiency and performance of existing sparse models like X-Nets, addressing hardware limitations in deep learning.

Contribution

The authors propose a novel deterministic method for creating diverse sparse neural network topologies without relying on dense references or pruning, enhancing the variety of sparse models.

Findings

Generated topologies are more diverse than X-Nets.

Sparse topologies preserve training performance.

Algorithm improves hardware efficiency for training deep networks.

Abstract

The sizes of deep neural networks (DNNs) are rapidly outgrowing the capacity of hardware to store and train them. Research over the past few decades has explored the prospect of sparsifying DNNs before, during, and after training by pruning edges from the underlying topology. The resulting neural network is known as a sparse neural network. More recent work has demonstrated the remarkable result that certain sparse DNNs can train to the same precision as dense DNNs at lower runtime and storage cost. An intriguing class of these sparse DNNs is the X-Nets, which are initialized and trained upon a sparse topology with neither reference to a parent dense DNN nor subsequent pruning. We present an algorithm that deterministically generates sparse DNN topologies that, as a whole, are much more diverse than X-Net topologies, while preserving X-Nets' desired characteristics.