One Leaf Reveals the Season: Occlusion-Based Contrastive Learning with Semantic-Aware Views for Efficient Visual Representation

TL;DR

This paper introduces occluded image contrastive learning (OCL), a simple and scalable pre-training method that masks image patches to generate diverse semantic views, enabling efficient high-level visual representation learning without complex augmentations.

Contribution

The paper presents a novel occlusion-based contrastive learning framework that emphasizes semantic-level view generation and high-level feature extraction, improving efficiency and scalability over existing methods.

Findings

OCL achieves 85.8% accuracy on downstream tasks.

Pre-training with ViT-L/16 takes only 133 hours on 4 GPUs.

OCL outperforms traditional augmentation-based contrastive methods.

Abstract

This paper proposes a scalable and straightforward pre-training paradigm for efficient visual conceptual representation called occluded image contrastive learning (OCL). Our OCL approach is simple: we randomly mask patches to generate different views within an image and contrast them among a mini-batch of images. The core idea behind OCL consists of two designs. First, masked tokens have the potential to significantly diminish the conceptual redundancy inherent in images, and create distinct views with substantial fine-grained differences on the semantic concept level instead of the instance level. Second, contrastive learning is adept at extracting high-level semantic conceptual features during the pre-training, circumventing the high-frequency interference and additional costs associated with image reconstruction. Importantly, OCL learns highly semantic conceptual representations efficiently without relying on hand-crafted data augmentations or additional auxiliary modules. Empirically, OCL demonstrates high scalability with Vision Transformers, as the ViT-L/16 can complete pre-training in 133 hours using only 4 A100 GPUs, achieving 85.8\% accuracy in downstream fine-tuning tasks. Code is available at https://anonymous.4open.science/r/OLRS/.