HIFICL: High-Fidelity In-Context Learning for Multimodal Tasks

TL;DR

HIFICL introduces a novel method for more accurately modeling in-context learning in multimodal models, leading to improved performance on benchmarks by using virtual key-value pairs and low-rank factorization.

Contribution

It proposes a new approach, HIFICL, that better captures the ICL mechanism in multimodal models through learnable context and efficient training techniques.

Findings

HIFICL outperforms existing approximation methods on multimodal benchmarks.

The method effectively models the influence of demonstrations in ICL.

Code is publicly available at the provided GitHub link.

Abstract

In-Context Learning (ICL) is a significant paradigm for Large Multimodal Models (LMMs), using a few in-context demonstrations (ICDs) for new task adaptation. However, its performance is sensitive to demonstration configurations and computationally expensive. Mathematically, the influence of these demonstrations can be decomposed into a dynamic mixture of the standard attention output and the context values. Current approximation methods simplify this process by learning a "shift vector". Inspired by the exact decomposition, we introduce High-Fidelity In-Context Learning (HIFICL) to more faithfully model the ICL mechanism. HIFICL consists of three key components: 1) a set of "virtual key-value pairs" to act as a learnable context, 2) a low-rank factorization for stable and regularized training, and 3) a simple end-to-end training objective. From another perspective, this mechanism constitutes a form of context-aware Parameter-Efficient Fine-Tuning (PEFT). Extensive experiments show that HiFICL consistently outperforms existing approximation methods on several multimodal benchmarks. The code is available at https://github.com/bbbandari/HiFICL.