ClosNets: a Priori Sparse Topologies for Faster DNN Training

TL;DR

ClosNets introduce a novel sparse topology for fully-connected layers in DNNs, enabling smaller, faster training without accuracy loss by using pre-selected sparse matrices, thus reducing memory and computation requirements.

Contribution

This work proposes a new fully-connected layer design with a priori sparse topologies, allowing training of smaller networks with reduced memory and computation without sacrificing accuracy.

Findings

ClosNets reduce dense layer sizes by up to an order of magnitude.

They achieve significant training speedups due to smaller network size.

High path diversity and shallowness contribute to performance.

Abstract

Fully-connected layers in deep neural networks (DNN) are often the throughput and power bottleneck during training. This is due to their large size and low data reuse. Pruning dense layers can significantly reduce the size of these networks, but this approach can only be applied after training. In this work we propose a novel fully-connected layer that reduces the memory requirements of DNNs without sacrificing accuracy. We replace a dense matrix with products of sparse matrices whose topologies we pick in advance. This allows us to: (1) train significantly smaller networks without a loss in accuracy, and (2) store the network weights without having to store connection indices. We therefore achieve significant training speedups due to the smaller network size, and a reduced amount of computation per epoch. We tested several sparse layer topologies and found that Clos networks perform well due to their high path diversity, shallowness, and high model accuracy. With the ClosNets, we are able to reduce dense layer sizes by as much as an order of magnitude without hurting model accuracy.