AMPS: Adaptive Modality Preference Steering via Functional Entropy

TL;DR

This paper introduces AMPS, a method for dynamically adjusting modality preference in multimodal large language models using an instance-aware metric and learnable scaling, improving control and reducing errors.

Contribution

It presents a novel instance-aware steering approach that adaptively modulates modality preference based on sample-specific information, outperforming uniform steering methods.

Findings

Outperforms traditional uniform steering in controlling modality preference.

Reduces generation error rates while maintaining effective modality adjustment.

Demonstrates improved sample-specific steering effectiveness.

Abstract

Multimodal Large Language Models (MLLMs) often exhibit significant modality preference, which is a tendency to favor one modality over another. Depending on the input, they may over-rely on linguistic priors relative to visual evidence, or conversely over-attend to visually salient but facts in textual contexts. Prior work has applied a uniform steering intensity to adjust the modality preference of MLLMs. However, strong steering can impair standard inference and increase error rates, whereas weak steering is often ineffective. In addition, because steering sensitivity varies substantially across multimodal instances, a single global strength is difficult to calibrate. To address this limitation with minimal disruption to inference, we introduce an instance-aware diagnostic metric that quantifies each modality's information contribution and reveals sample-specific susceptibility to steering. Building on these insights, we propose a scaling strategy that reduces steering for sensitive samples and a learnable module that infers scaling patterns, enabling instance-aware control of modality preference. Experimental results show that our instance-aware steering outperforms conventional steering in modulating modality preference, achieving effective adjustment while keeping generation error rates low.