The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models

TL;DR

This paper investigates whether highly sparse subnetworks, identified via the lottery ticket hypothesis, can be found in pre-trained computer vision models and still perform well on various downstream tasks, regardless of the pre-training method.

Contribution

It extends the lottery ticket hypothesis to pre-trained vision models, demonstrating the existence of effective sparse subnetworks that maintain performance across multiple tasks.

Findings

Matching subnetworks exist at high sparsity levels (59-96%) in pre-trained models.

Subnetwork performance remains comparable to full models on downstream tasks.

Diverse mask structures and sensitivities are observed across different pre-training methods.

Abstract

The computer vision world has been re-gaining enthusiasm in various pre-trained models, including both classical ImageNet supervised pre-training and recently emerged self-supervised pre-training such as simCLR and MoCo. Pre-trained weights often boost a wide range of downstream tasks including classification, detection, and segmentation. Latest studies suggest that pre-training benefits from gigantic model capacity. We are hereby curious and ask: after pre-training, does a pre-trained model indeed have to stay large for its downstream transferability? In this paper, we examine supervised and self-supervised pre-trained models through the lens of the lottery ticket hypothesis (LTH). LTH identifies highly sparse matching subnetworks that can be trained in isolation from (nearly) scratch yet still reach the full models' performance. We extend the scope of LTH and question whether matching subnetworks still exist in pre-trained computer vision models, that enjoy the same downstream transfer performance. Our extensive experiments convey an overall positive message: from all pre-trained weights obtained by ImageNet classification, simCLR, and MoCo, we are consistently able to locate such matching subnetworks at 59.04% to 96.48% sparsity that transfer universally to multiple downstream tasks, whose performance see no degradation compared to using full pre-trained weights. Further analyses reveal that subnetworks found from different pre-training tend to yield diverse mask structures and perturbation sensitivities. We conclude that the core LTH observations remain generally relevant in the pre-training paradigm of computer vision, but more delicate discussions are needed in some cases. Codes and pre-trained models will be made available at: https://github.com/VITA-Group/CV_LTH_Pre-training.