Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Models

TL;DR

This paper introduces UE-DPO, a novel method that uses token-level epistemic uncertainty to guide learning in multimodal large language models, improving visual grounding and reducing hallucinations.

Contribution

It proposes an uncertainty-aware exploration strategy for direct preference optimization, enabling models to identify and correct their visual grounding deficiencies.

Findings

UE-DPO improves visual grounding accuracy in MLLMs.

The method enhances robustness against hallucinations.

Experiments validate the theoretical benefits of uncertainty-guided exploration.

Abstract

Direct Preference Optimization (DPO) has proven to be an effective solution for mitigating hallucination in Multimodal Large Language Models (MLLMs) by learning from preference pairs. One of its key challenges lies in how to transfer the sequence-level preference into fine-grained supervision on visual fidelity. To safeguard vision-related tokens that are prone to hallucination, existing methods typically allocate training emphasis according to the model's self-assessed visual sensitivity signals. However, such sensitivity, estimated by a model still under training, introduces self-referential bias: reinforcing already well-learned visual cues while neglecting hard-to-perceive but critical details, thereby limiting deeper alignment. In this work, we propose an Uncertainty-aware Exploratory Direct Preference Optimization (UE-DPO) method for MLLMs, which enables the model to uncover its cognitive deficiencies and actively explore for self-correction, guided by token-level epistemic uncertainty. Specifically, we first quantify the uncertainty from the model's failure to ground token predictions in the given image. Then, based on an uncertainty-aware exploration intensity, we encourage more learning pressure on visually deficient tokens in preferred samples, and alleviate the over-penalization of beneficial knowledge in dispreferred samples. Further, we provide a theoretical justification for our method, and extensive experiments demonstrate its effectiveness and robustness.