Revisiting Loss Modelling for Unstructured Pruning

TL;DR

This paper reevaluates loss modeling techniques for unstructured neural network pruning, emphasizing the importance of local pruning steps and comparing first and second order Taylor expansions, revealing that preserving the original network function doesn't always lead to better post-pruning performance.

Contribution

It systematically compares first and second order Taylor expansions for pruning and highlights the significance of locality in pruning steps, challenging existing assumptions about loss preservation.

Findings

Both first and second order Taylor expansions achieve similar pruning performance.

Locality in pruning steps is crucial for effective unstructured pruning.

Preserving the original network function does not guarantee better fine-tuned performance.

Abstract

By removing parameters from deep neural networks, unstructured pruning methods aim at cutting down memory footprint and computational cost, while maintaining prediction accuracy. In order to tackle this otherwise intractable problem, many of these methods model the loss landscape using first or second order Taylor expansions to identify which parameters can be discarded. We revisit loss modelling for unstructured pruning: we show the importance of ensuring locality of the pruning steps. We systematically compare first and second order Taylor expansions and empirically show that both can reach similar levels of performance. Finally, we show that better preserving the original network function does not necessarily transfer to better performing networks after fine-tuning, suggesting that only considering the impact of pruning on the loss might not be a sufficient objective to design good pruning criteria.