MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization

TL;DR

This paper introduces MoD-DPO, a framework that enhances the reliability of omni-modal large language models by reducing cross-modal hallucinations through modality-aware regularization and language-prior debiasing.

Contribution

The paper presents a novel modality-decoupled preference optimization method that improves modality grounding and reduces hallucinations in omni LLMs.

Findings

MoD-DPO outperforms previous methods on audiovisual hallucination benchmarks.

It improves perception accuracy and hallucination resistance.

The approach demonstrates scalable enhancement of multimodal model reliability.

Abstract

Omni-modal large language models (omni LLMs) have recently achieved strong performance across audiovisual understanding tasks, yet they remain highly susceptible to cross-modal hallucinations arising from spurious correlations and dominant language priors. In this work, we propose Modality-Decoupled Direct Preference Optimization (MoD-DPO), a simple and effective framework for improving modality grounding in omni LLMs. MoD-DPO introduces modality-aware regularization terms that explicitly enforce invariance to corruptions in irrelevant modalities and sensitivity to perturbations in relevant modalities, thereby reducing unintended cross-modal interactions. To further mitigate over-reliance on textual priors, we incorporate a language-prior debiasing penalty that discourages hallucination-prone text-only responses. Extensive experiments across multiple audiovisual hallucination benchmarks demonstrate that MoD-DPO consistently improves perception accuracy and hallucination resistance, outperforming previous preference optimization baselines under similar training budgets. Our findings underscore the importance of modality-faithful alignment and demonstrate a scalable path toward more reliable and resilient multimodal foundation models.