Is Oracle Pruning the True Oracle?

TL;DR

This study critically examines the validity of oracle pruning in modern deep learning models, revealing that weights selected based on initial performance do not reliably predict post-retraining performance, challenging longstanding assumptions.

Contribution

The paper empirically tests the correlation between pre- and post-retraining performance in oracle pruning, demonstrating its unreliability on modern models and datasets, and highlights the need to consider retraining in pruning criteria.

Findings

Pre- and post-retraining performance are barely correlated.

Oracle pruning weights do not guarantee good post-retraining performance.

Task complexity affects the validity of oracle pruning.

Abstract

Oracle pruning, which selects unimportant weights by minimizing the pruned train loss, has been taken as the foundation for most neural network pruning methods for over 35 years, while few (if not none) have thought about how much the foundation really holds. This paper, for the first time, attempts to examine its validity on modern deep models through empirical correlation analyses and provide reflections on the field of neural network pruning. Specifically, for a typical pruning algorithm with three stages (pertaining, pruning, and retraining), we analyze the model performance correlation before and after retraining. Extensive experiments (37K models are trained) across a wide spectrum of models (LeNet5, VGG, ResNets, ViT, MLLM) and datasets (MNIST and its variants, CIFAR10/CIFAR100, ImageNet-1K, MLLM data) are conducted. The results lead to a surprising conclusion: on modern deep learning models, the performance before retraining is barely correlated with the performance after retraining. Namely, the weights selected by oracle pruning can hardly guarantee a good performance after retraining. This further implies that existing works using oracle pruning to derive pruning criteria may be groundless from the beginning. Further studies suggest the rising task complexity is one factor that makes oracle pruning invalid nowadays. Finally, given the evidence, we argue that the retraining stage in a pruning algorithm should be accounted for when developing any pruning criterion.