CrossTransformers: spatially-aware few-shot transfer

TL;DR

This paper introduces CrossTransformers, a novel Transformer-based architecture combined with self-supervised learning to improve few-shot transfer in vision systems, addressing supervision collapse and enhancing robustness across tasks and domains.

Contribution

The paper proposes CrossTransformers and a self-supervised training approach to mitigate supervision collapse, enabling more effective few-shot transfer in vision models.

Findings

Achieved state-of-the-art results on Meta-Dataset for transfer learning.

Demonstrated robustness to task and domain shifts.

Showed improved generalization with spatially-aware features.

Abstract

Given new tasks with very little data$-$such as new classes in a classification problem or a domain shift in the input$-$performance of modern vision systems degrades remarkably quickly. In this work, we illustrate how the neural network representations which underpin modern vision systems are subject to supervision collapse, whereby they lose any information that is not necessary for performing the training task, including information that may be necessary for transfer to new tasks or domains. We then propose two methods to mitigate this problem. First, we employ self-supervised learning to encourage general-purpose features that transfer better. Second, we propose a novel Transformer based neural network architecture called CrossTransformers, which can take a small number of labeled images and an unlabeled query, find coarse spatial correspondence between the query and the labeled images, and then infer class membership by computing distances between spatially-corresponding features. The result is a classifier that is more robust to task and domain shift, which we demonstrate via state-of-the-art performance on Meta-Dataset, a recent dataset for evaluating transfer from ImageNet to many other vision datasets.