Re-Weighted Learning for Sparsifying Deep Neural Networks

TL;DR

This paper introduces a novel affine scaling transformation algorithm to sparsify deep neural networks, enabling simultaneous layer-wise sparse connection learning without performance loss, addressing storage and computation issues.

Contribution

It proposes a general affine scaling transformation framework for sparsifying DNNs, allowing for simultaneous layer-wise sparse connection learning, which is a novel approach in this context.

Findings

Achieves comparable pruning results to existing methods.

Learns sparse connections at each layer simultaneously.

Addresses storage and computational challenges of DNNs.

Abstract

This paper addresses the topic of sparsifying deep neural networks (DNN's). While DNN's are powerful models that achieve state-of-the-art performance on a large number of tasks, the large number of model parameters poses serious storage and computational challenges. To combat these difficulties, a growing line of work focuses on pruning network weights without sacrificing performance. We propose a general affine scaling transformation (AST) algorithm to sparsify DNN's. Our approach follows in the footsteps of popular sparse recovery techniques, which have yet to be explored in the context of DNN's. We describe a principled framework for transforming densely connected DNN's into sparsely connected ones without sacrificing network performance. Unlike existing methods, our approach is able to learn sparse connections at each layer simultaneously, and achieves comparable pruning results on the architecture tested.