Where and What Matters: Sensitivity-Aware Task Vectors for Many-Shot Multimodal In-Context Learning

TL;DR

This paper introduces a sensitivity-aware framework for inserting task vectors into large multimodal models, improving in-context learning by identifying optimal insertion points and values through structural activation patterns and reinforcement learning.

Contribution

It proposes a novel method to determine where and what to insert as task vectors in multimodal models, enhancing performance over previous approaches.

Findings

Consistent performance improvements across multiple models and tasks.

Effective identification of sensitive insertion locations using activation patterns.

Robust generalization of the proposed method.

Abstract

Large Multimodal Models (LMMs) have shown promising in-context learning (ICL) capabilities, but scaling to many-shot settings remains difficult due to limited context length and high inference cost. To address these challenges, task-vector-based methods have been explored by inserting compact representations of many-shot in-context demonstrations into model activations. However, existing task-vector-based methods either overlook the importance of where to insert task vectors or struggle to determine suitable values for each location. To this end, we propose a novel Sensitivity-aware Task Vector insertion framework (STV) to figure out where and what to insert. Our key insight is that activation deltas across query-context pairs exhibit consistent structural patterns, providing a reliable cue for insertion. Based on the identified sensitive-aware locations, we construct a pre-clustered activation bank for each location by clustering the activation values, and then apply reinforcement learning to choose the most suitable one to insert. We evaluate STV across a range of multimodal models (e.g., Qwen-VL, Idefics-2) and tasks (e.g., VizWiz, OK-VQA), demonstrating its effectiveness and showing consistent improvements over previous task-vector-based methods with strong generalization.