Weight Reparametrization for Budget-Aware Network Pruning

TL;DR

This paper introduces a novel end-to-end network pruning method that learns lightweight architectures through reparametrization, eliminating the need for fine-tuning and enabling efficient, budget-aware pruning.

Contribution

The proposed method jointly trains and prunes networks using reparametrization to implicitly learn pruning masks without increasing training parameters.

Findings

Effective pruning on CIFAR10 and TinyImageNet datasets

Achieves high accuracy without fine-tuning

Compatible with standard architectures like VGG19 and ResNet18

Abstract

Pruning seeks to design lightweight architectures by removing redundant weights in overparameterized networks. Most of the existing techniques first remove structured sub-networks (filters, channels,...) and then fine-tune the resulting networks to maintain a high accuracy. However, removing a whole structure is a strong topological prior and recovering the accuracy, with fine-tuning, is highly cumbersome. In this paper, we introduce an "end-to-end" lightweight network design that achieves training and pruning simultaneously without fine-tuning. The design principle of our method relies on reparametrization that learns not only the weights but also the topological structure of the lightweight sub-network. This reparametrization acts as a prior (or regularizer) that defines pruning masks implicitly from the weights of the underlying network, without increasing the number of training parameters. Sparsity is induced with a budget loss that provides an accurate pruning. Extensive experiments conducted on the CIFAR10 and the TinyImageNet datasets, using standard architectures (namely Conv4, VGG19 and ResNet18), show compelling results without fine-tuning.