$\Delta$-AttnMask: Attention-Guided Masked Hidden States for Efficient Data Selection and Augmentation

TL;DR

$ Delta$-AttnMask is a novel, efficient data selection method for Vision-Language Model fine-tuning that evaluates sample quality through attention-guided masking, reducing data needs and improving accuracy.

Contribution

It introduces a model-agnostic, data-agnostic framework that assesses sample quality via loss differences, enabling effective data selection without extra labels or training.

Findings

Achieves state-of-the-art performance with only 20% of data.

Accelerates training by 5 times compared to full datasets.

Surpasses full-dataset baselines by +10.1% accuracy.

Abstract

Visual Instruction Finetuning (VIF) is pivotal for post-training Vision-Language Models (VLMs). Unlike unimodal instruction finetuning in plain-text large language models, which mainly requires instruction datasets to enable model instruction-following ability, VIF also requires multimodal data to enable joint visual and textual understanding; therefore, it typically requires more data. Consequently, VIF imposes stricter data selection challenges: the method must scale efficiently to handle larger data demands while ensuring the quality of both visual and textual content, as well as their alignment. Despite its critical impact on performance, data selection for VIF remains an understudied area. In this paper, we propose $\Delta$-AttnMask. This data-efficient framework quantifies sample quality through attention-guided masking of the model's hidden states, jointly evaluating image-text pairs without requiring domain labels, auxiliary models, or extra training. By computing loss differences ($\Delta$) between the original states and states masked using high-attention regions, $\Delta$-AttnMask intrinsically assesses sample quality. Experiments across multiple VLMs and datasets show that $\Delta$-AttnMask achieves state-of-the-art performance with just 20% of data, accelerating training by 5x while surpassing full-dataset baselines by +10.1% in overall accuracy. Its model-agnostic and data-agnostic design ensures broad applicability across modalities and architectures.