IRIS: Implicit Reward-Guided Internal Sifting for Mitigating Multimodal Hallucination

TL;DR

IRIS introduces a novel on-policy method leveraging implicit rewards within the model's native space to effectively reduce hallucinations in multimodal large language models without external evaluators.

Contribution

The paper proposes IRIS, a new implicit reward-guided internal sifting approach that directly addresses modal conflicts and hallucinations in MLLMs using internal signals and self-generated preferences.

Findings

IRIS achieves competitive hallucination mitigation performance.

Uses only 5.7k samples without external feedback.

Effectively captures internal modal conflicts.

Abstract

Hallucination remains a fundamental challenge for Multimodal Large Language Models (MLLMs). While Direct Preference Optimization (DPO) is a key alignment framework, existing approaches often rely heavily on costly external evaluators for scoring or rewriting, incurring off-policy learnability gaps and discretization loss. Due to the lack of access to internal states, such feedback overlooks the fine-grained conflicts between different modalities that lead to hallucinations during generation. To address this issue, we propose IRIS (Implicit Reward-Guided Internal Sifting), which leverages continuous implicit rewards in the native log-probability space to preserve full information density and capture internal modal competition. This on-policy paradigm eliminates learnability gaps by utilizing self-generated preference pairs. By sifting these pairs based on multimodal implicit rewards, IRIS ensures that optimization is driven by signals that directly resolve modal conflicts. Extensive experiments demonstrate that IRIS achieves highly competitive performance on key hallucination benchmarks using only 5.7k samples, without requiring any external feedback during preference alignment. These results confirm that IRIS provides an efficient and principled paradigm for mitigating MLLM hallucinations.