DSPNet: Towards Slimmable Pretrained Networks based on Discriminative Self-supervised Learning

TL;DR

DSPNet introduces a slimmable, discriminative self-supervised learning framework that efficiently pretrains multiple network sizes simultaneously, achieving comparable or better downstream performance with reduced training costs.

Contribution

The paper presents a novel discriminative SSL approach for slimmable networks, enabling simultaneous pretraining of multiple sizes and improving efficiency and transferability.

Findings

DSPNet achieves comparable or better performance than individually pretrained networks.

Reduces training cost significantly compared to training multiple models separately.

Pretrained models generalize well to downstream detection and segmentation tasks.

Abstract

Self-supervised learning (SSL) has achieved promising downstream performance. However, when facing various resource budgets in real-world applications, it costs a huge computation burden to pretrain multiple networks of various sizes one by one. In this paper, we propose Discriminative-SSL-based Slimmable Pretrained Networks (DSPNet), which can be trained at once and then slimmed to multiple sub-networks of various sizes, each of which faithfully learns good representation and can serve as good initialization for downstream tasks with various resource budgets. Specifically, we extend the idea of slimmable networks to a discriminative SSL paradigm, by integrating SSL and knowledge distillation gracefully. We show comparable or improved performance of DSPNet on ImageNet to the networks individually pretrained one by one under the linear evaluation and semi-supervised evaluation protocols, while reducing large training cost. The pretrained models also generalize well on downstream detection and segmentation tasks. Code will be made public.