# PruneTrain: Fast Neural Network Training by Dynamic Sparse Model Reconfiguration

## TL;DR

PruneTrain is a dynamic, structured pruning method that accelerates CNN training by reducing computation, memory, and communication costs while maintaining high accuracy, demonstrated on ResNet50 with significant training time reduction.

## Contribution

It introduces a novel structured group-lasso regularization and reconfiguration technique to efficiently prune and reconfigure neural networks during training.

## Key findings

- Achieves 39% reduction in training time for ResNet50 on ImageNet.
- Reduces FLOPs by 40% and memory bandwidth by 37%.
- Decreases inter-accelerator communication by 55%.

## Abstract

State-of-the-art convolutional neural networks (CNNs) used in vision applications have large models with numerous weights. Training these models is very compute- and memory-resource intensive. Much research has been done on pruning or compressing these models to reduce the cost of inference, but little work has addressed the costs of training. We focus precisely on accelerating training. We propose PruneTrain, a cost-efficient mechanism that gradually reduces the training cost during training. PruneTrain uses a structured group-lasso regularization approach that drives the training optimization toward both high accuracy and small weight values. Small weights can then be periodically removed by reconfiguring the network model to a smaller one. By using a structured-pruning approach and additional reconfiguration techniques we introduce, the pruned model can still be efficiently processed on a GPU accelerator. Overall, PruneTrain achieves a reduction of 39% in the end-to-end training time of ResNet50 for ImageNet by reducing computation cost by 40% in FLOPs, memory accesses by 37% for memory bandwidth bound layers, and the inter-accelerator communication by 55%.