Self-Masking Networks for Unsupervised Adaptation

TL;DR

This paper introduces Self-Masking Networks (SMNs), a self-supervised fine-tuning method using binary masks that significantly enhances performance and efficiency when adapting large models to downstream computer vision tasks with limited labeled data.

Contribution

The paper presents a novel self-supervised masking approach for efficient model adaptation, demonstrating substantial improvements in label-efficient scenarios across multiple datasets and architectures.

Findings

SMNs are up to 79x more storage-efficient.

SMNs significantly improve downstream task performance.

Effective in 3 label-efficient settings.

Abstract

With the advent of billion-parameter foundation models, efficient fine-tuning has become increasingly important for the adaptation of models to downstream tasks. However, especially in computer vision, it can be hard to achieve good performance when access to quality labeled data is lacking. In this work, we propose a method adapting pretrained generalist models in a self-supervised manner by learning binary masks. These self-supervised masking networks (SMNs) are up to 79x more efficient to store and significantly improve performance on label-efficient downstream tasks. We validate the usefulness of learning binary masks as a fine-tuning method on 8 datasets and 3 model architectures, and we demonstrate the effectiveness of SMNs in 3 label-efficient settings.