MAST: Masked Augmentation Subspace Training for Generalizable Self-Supervised Priors

TL;DR

MAST introduces a novel self-supervised learning approach that disentangles feature representations into subspaces corresponding to different data augmentations, enhancing generalization across diverse downstream tasks.

Contribution

The paper proposes Masked Augmentation Subspace Training (MAST), a method that factorizes feature space into augmentation-specific subspaces for improved task-agnostic generalization.

Findings

MAST improves downstream task performance across classification, detection, and segmentation.

Disentangled subspaces effectively capture augmentation priors.

Uncertainty modeling helps reweight ambiguous samples for better learning.

Abstract

Recent Self-Supervised Learning (SSL) methods are able to learn feature representations that are invariant to different data augmentations, which can then be transferred to downstream tasks of interest. However, different downstream tasks require different invariances for their best performance, so the optimal choice of augmentations for SSL depends on the target task. In this paper, we aim to learn self-supervised features that generalize well across a variety of downstream tasks (e.g., object classification, detection and instance segmentation) without knowing any task information beforehand. We do so by Masked Augmentation Subspace Training (or MAST) to encode in the single feature space the priors from different data augmentations in a factorized way. Specifically, we disentangle the feature space into separate subspaces, each induced by a learnable mask that selects relevant feature dimensions to model invariance to a specific augmentation. We show the success of MAST in jointly capturing generalizable priors from different augmentations, using both unique and shared features across the subspaces. We further show that MAST benefits from uncertainty modeling to reweight ambiguous samples from strong augmentations that may cause similarity mismatch in each subspace. Experiments demonstrate that MAST consistently improves generalization on various downstream tasks, while being task-agnostic and efficient during SSL. We also provide interesting insights about how different augmentations are related and how uncertainty reflects learning difficulty.