Unsupervised Meta-Learning via In-Context Learning

TL;DR

This paper introduces a novel unsupervised meta-learning method that uses in-context learning with transformers to improve transferability to new tasks, achieving state-of-the-art results on benchmarks.

Contribution

It reframes unsupervised meta-learning as a sequence modeling problem using transformers, enabling better generalization through diverse task creation strategies.

Findings

Outperforms existing unsupervised meta-learning baselines.

Achieves results comparable to supervised and self-supervised methods.

Demonstrates the effectiveness of in-context learning for meta-learning.

Abstract

Unsupervised meta-learning aims to learn feature representations from unsupervised datasets that can transfer to downstream tasks with limited labeled data. In this paper, we propose a novel approach to unsupervised meta-learning that leverages the generalization abilities of in-context learning observed in transformer architectures. Our method reframes meta-learning as a sequence modeling problem, enabling the transformer encoder to learn task context from support images and utilize it to predict query images. At the core of our approach lies the creation of diverse tasks generated using a combination of data augmentations and a mixing strategy that challenges the model during training while fostering generalization to unseen tasks at test time. Experimental results on benchmark datasets showcase the superiority of our approach over existing unsupervised meta-learning baselines, establishing it as the new state-of-the-art. Remarkably, our method achieves competitive results with supervised and self-supervised approaches, underscoring its efficacy in leveraging generalization over memorization.