Unsqueeze [CLS] Bottleneck to Learn Rich Representations

TL;DR

UDI is a novel self-supervised learning method that enhances representation richness by preserving input information, leading to improved performance in classification, detection, and segmentation tasks.

Contribution

UDI introduces a stratified sampling-based distillation approach to retain more input information in self-supervised learning, improving downstream task performance.

Findings

UDI achieves state-of-the-art or competitive results in image classification.

UDI significantly improves dense prediction tasks like object detection and segmentation.

UDI enhances low-shot learning capabilities.

Abstract

Distillation-based self-supervised learning typically leads to more compressed representations due to its radical clustering process and the implementation of a sharper target distribution. To overcome this limitation and preserve more information from input, we introduce UDI, conceptualized as Unsqueezed Distillation-based self-supervised learning (SSL). UDI enriches the learned representation by encouraging multimodal prediction distilled from a consolidated profile of local predictions that are derived via stratified sampling. Our evaluations show that UDI not only promotes semantically meaningful representations at instance level, delivering superior or competitive results to state-of-the-art SSL methods in image classification, but also effectively preserves the nuisance of input, which yields significant improvement in dense prediction tasks, including object detection and segmentation. Additionally, UDI performs competitively in low-shot image classification, improving the scalability of joint-embedding pipelines. Various visualizations and ablation studies are presented to further elucidate the mechanisms behind UDI. Our source code is available at https://github.com/ISL-CV/udi.