Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning

TL;DR

This paper introduces Multimodal Task Vectors (MTV), a method to enable large multimodal models to perform many-shot in-context learning by compressing multiple examples into implicit representations, overcoming context length limitations.

Contribution

The paper proposes a novel approach using MTV to enable many-shot multimodal in-context learning without finetuning or increasing context length.

Findings

MTV exists in large multimodal models and can be extracted.

MTV enables effective many-shot in-context learning across vision-and-language tasks.

Performance scales with the number of compressed shots and generalizes to out-of-domain tasks.

Abstract

The recent success of interleaved Large Multimodal Models (LMMs) in few-shot learning suggests that in-context learning (ICL) with many examples can be promising for learning new tasks. However, this many-shot multimodal ICL setting has one crucial problem: it is fundamentally limited by the model's context length set at pretraining. The problem is especially prominent in the multimodal domain, which processes both text and images, requiring additional tokens. This motivates the need for a multimodal method to compress many shots into fewer tokens without finetuning. In this work, we enable LMMs to perform multimodal, many-shot in-context learning by leveraging Multimodal Task Vectors (MTV) -- compact implicit representations of in-context examples compressed in the model's attention heads. Specifically, we first demonstrate the existence of such MTV in LMMs and then leverage these extracted MTV to enable many-shot in-context learning for various vision-and-language tasks. Our experiments suggest that MTV can scale in performance with the number of compressed shots and generalize to similar out-of-domain tasks without additional context length for inference. Code: https://github.com/Brandon3964/MultiModal-Task-Vector