COPO: Causal-Oriented Policy Optimization for Hallucinations of MLLMs

TL;DR

This paper introduces COPO, a causal-oriented policy optimization method that reduces hallucinations in multimodal large language models by focusing on causally relevant tokens, leading to more accurate and grounded outputs.

Contribution

The paper proposes a novel causal-oriented optimization framework that explicitly constrains token relevance to mitigate hallucinations in MLLMs, a significant advancement over existing methods.

Findings

COPO effectively reduces hallucinations in MLLMs.

Experimental results show improved factual accuracy and grounding.

The method outperforms baseline models on multiple benchmarks.

Abstract

Despite Multimodal Large Language Models (MLLMs) having shown impressive capabilities, they may suffer from hallucinations. Empirically, we find that MLLMs attend disproportionately to task-irrelevant background regions compared with text-only LLMs, implying spurious background-answer correlations. We claim and analyze that (i) outcome-based rewards can be an important factor leading to spurious correlations, and (ii) spurious correlations can be an important factor leading to hallucinations. Based on these results, we propose Causal-Oriented Policy Optimization (COPO) to mitigate these spurious correlations, thus addressing the issue of hallucinations. It imposes token-level sufficiency and necessity constraints to measure each inference token's causal contribution, thus ensuring correct and evidence-grounded output. Specifically, we first evaluate each token's causal contribution via a newly proposed causal completeness reward. This reward is then used to construct a causally informed advantage function within the GRPO optimization framework, encouraging the model to focus on tokens that are causally sufficient and necessary for accurate generation. Experimental results across various benchmarks demonstrate the advantages of COPO.