Improving self-supervised representation learning via sequential adversarial masking

TL;DR

This paper introduces a sequential adversarial masking framework for self-supervised learning in computer vision, enhancing the difficulty of reconstruction tasks and improving downstream task performance.

Contribution

It proposes a novel sequential adversarial masking method with different adversary constraints, advancing the state-of-the-art in masking-based SSL for vision tasks.

Findings

Improved classification accuracy on ImageNet100, STL10, CIFAR10/100

Enhanced segmentation performance on Pascal VOC

Demonstrated the effectiveness of adversarial masking in SSL

Abstract

Recent methods in self-supervised learning have demonstrated that masking-based pretext tasks extend beyond NLP, serving as useful pretraining objectives in computer vision. However, existing approaches apply random or ad hoc masking strategies that limit the difficulty of the reconstruction task and, consequently, the strength of the learnt representations. We improve upon current state-of-the-art work in learning adversarial masks by proposing a new framework that generates masks in a sequential fashion with different constraints on the adversary. This leads to improvements in performance on various downstream tasks, such as classification on ImageNet100, STL10, and CIFAR10/100 and segmentation on Pascal VOC. Our results further demonstrate the promising capabilities of masking-based approaches for SSL in computer vision.