NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm

TL;DR

NeST is a neural network synthesis tool that combines growth and pruning algorithms to automatically generate compact, accurate DNN architectures, significantly reducing parameters and FLOPs across various models.

Contribution

The paper introduces a novel grow-and-prune paradigm for DNN synthesis, enabling automatic architecture optimization during training.

Findings

Achieves up to 74.3x parameter reduction on LeNet-5

Reduces FLOPs by up to 79.4x on LeNet-5

Outperforms pruning-only methods in compactness and accuracy

Abstract

Deep neural networks (DNNs) have begun to have a pervasive impact on various applications of machine learning. However, the problem of finding an optimal DNN architecture for large applications is challenging. Common approaches go for deeper and larger DNN architectures but may incur substantial redundancy. To address these problems, we introduce a network growth algorithm that complements network pruning to learn both weights and compact DNN architectures during training. We propose a DNN synthesis tool (NeST) that combines both methods to automate the generation of compact and accurate DNNs. NeST starts with a randomly initialized sparse network called the seed architecture. It iteratively tunes the architecture with gradient-based growth and magnitude-based pruning of neurons and connections. Our experimental results show that NeST yields accurate, yet very compact DNNs, with a wide range of seed architecture selection. For the LeNet-300-100 (LeNet-5) architecture, we reduce network parameters by 70.2x (74.3x) and floating-point operations (FLOPs) by 79.4x (43.7x). For the AlexNet and VGG-16 architectures, we reduce network parameters (FLOPs) by 15.7x (4.6x) and 30.2x (8.6x), respectively. NeST's grow-and-prune paradigm delivers significant additional parameter and FLOPs reduction relative to pruning-only methods.