Towards Few-Shot Adaptation of Foundation Models via Multitask Finetuning

TL;DR

This paper provides a theoretical and empirical study on how multitask finetuning of foundation models with diverse related tasks improves adaptation to new, low-label target tasks, and proposes a task selection algorithm.

Contribution

It offers a theoretical analysis linking task diversity and consistency to improved target task performance and introduces a practical task selection method.

Findings

Multitask finetuning reduces error on target tasks compared to direct adaptation.

Diversity and consistency metrics predict finetuning effectiveness.

The proposed task selection algorithm improves model performance on target tasks.

Abstract

Foundation models have emerged as a powerful tool for many AI problems. Despite the tremendous success of foundation models, effective adaptation to new tasks, particularly those with limited labels, remains an open question and lacks theoretical understanding. An emerging solution with recent success in vision and NLP involves finetuning a foundation model on a selection of relevant tasks, before its adaptation to a target task with limited labeled samples. In this paper, we study the theoretical justification of this multitask finetuning approach. Our theoretical analysis reveals that with a diverse set of related tasks, this multitask finetuning leads to reduced error in the target task, in comparison to directly adapting the same pretrained model. We quantify the relationship between finetuning tasks and target tasks by diversity and consistency metrics, and further propose a practical task selection algorithm. We substantiate our theoretical claims with extensive empirical evidence. Further, we present results affirming our task selection algorithm adeptly chooses related finetuning tasks, providing advantages to the model performance on target tasks. We believe our study shed new light on the effective adaptation of foundation models to new tasks that lack abundant labels. Our code is available at https://github.com/OliverXUZY/Foudation-Model_Multitask.