Rethinking Task Sampling for Few-shot Vision-Language Transfer Learning

TL;DR

This paper introduces a simple task sampling strategy called MAMF that improves few-shot vision-language transfer learning, outperforming classical fine-tuning across multiple tasks by focusing on effective task selection.

Contribution

The paper highlights the importance of task sampling in few-shot learning and proposes MAMF, a straightforward algorithm that enhances transfer performance without complex optimization.

Findings

MAMF outperforms classical fine-tuning on five tasks.

Task sampling significantly impacts few-shot transfer success.

Bi-level optimization in MAML is sensitive to zero-shot task performance.

Abstract

Despite achieving state-of-the-art zero-shot performance, existing vision-language models still fall short of few-shot transfer ability on domain-specific problems. Classical fine-tuning often fails to prevent highly expressive models from exploiting spurious correlations. Although model-agnostic meta-learning (MAML) presents as a natural alternative for few-shot transfer learning, the expensive computation due to implicit second-order optimization limits its use on large-scale vision-language models such as CLIP. While much literature has been devoted to exploring alternative optimization strategies, we identify another essential aspect towards effective few-shot transfer learning, task sampling, which is previously only be viewed as part of data pre-processing in MAML. To show the impact of task sampling, we propose a simple algorithm, Model-Agnostic Multitask Fine-tuning (MAMF), which differentiates classical fine-tuning only on uniformly sampling multiple tasks. Despite its simplicity, we show that MAMF consistently outperforms classical fine-tuning on five few-shot vision-language classification tasks. We further show that the effectiveness of the bi-level optimization in MAML is highly sensitive to the zero-shot performance of a task in the context of few-shot vision-language classification. The goal of this paper is to provide new insights on what makes few-shot learning work, and encourage more research into investigating better task sampling strategies.