Reconstruction Task Finds Universal Winning Tickets

TL;DR

This paper introduces a novel pruning method that incorporates image reconstruction to produce more transferable and effective pruned neural networks for diverse downstream tasks in computer vision.

Contribution

The paper proposes integrating pixel-level image reconstruction into pruning to enhance transferability of pruned models across various downstream tasks.

Findings

Outperforms state-of-the-art pruning methods on benchmark tasks.

Improves transferability of pruned models to complex downstream tasks.

Demonstrates effectiveness of reconstruction-based pruning in computer vision.

Abstract

Pruning well-trained neural networks is effective to achieve a promising accuracy-efficiency trade-off in computer vision regimes. However, most of existing pruning algorithms only focus on the classification task defined on the source domain. Different from the strong transferability of the original model, a pruned network is hard to transfer to complicated downstream tasks such as object detection arXiv:arch-ive/2012.04643. In this paper, we show that the image-level pretrain task is not capable of pruning models for diverse downstream tasks. To mitigate this problem, we introduce image reconstruction, a pixel-level task, into the traditional pruning framework. Concretely, an autoencoder is trained based on the original model, and then the pruning process is optimized with both autoencoder and classification losses. The empirical study on benchmark downstream tasks shows that the proposed method can outperform state-of-the-art results explicitly.