Optimizing LVLMs with On-Policy Data for Effective Hallucination Mitigation

TL;DR

This paper introduces a novel approach to reduce hallucinations in large vision-language models by leveraging on-policy data, a hallucination classifier, and a dynamic preference optimization algorithm, significantly improving model reliability.

Contribution

The authors propose an effective method for hallucination mitigation in LVLMs using on-policy data, a binary hallucination classifier, and an iterative preference optimization scheme, outperforming existing methods.

Findings

Halts hallucination rate by over 50% on MMHalBench

Reduces hallucination by 79.5% on Object HalBench

Enables open-source LVLMs to surpass GPT-4V performance

Abstract

Recently, large vision-language models (LVLMs) have risen to be a promising approach for multimodal tasks. However, principled hallucination mitigation remains a critical challenge.In this work, we first analyze the data generation process in LVLM hallucination mitigation and affirm that on-policy data significantly outperforms off-policy data, which thus calls for efficient and reliable preference annotation of on-policy data. We then point out that, existing annotation methods introduce additional hallucination in training samples, which may enhance the model's hallucination patterns, to address this problem, we propose training a hallucination classifier giving binary annotations, which guarantee clean chosen samples for the subsequent alignment. To further harness of the power of on-policy data, we design a robust iterative direct preference optimization (DPO) algorithm adopting a dynamic sample reweighting scheme. We conduct comprehensive experiments on three benchmarks with comparison to 8 state-of-the-art baselines. In particular, our approach reduces the hallucination rate of LLaVA-1.5-7B on MMHalBench by 50.8% and the average hallucination rate on Object HalBench by 79.5%; more significantly, our method fully taps into the potential of open-source models, enabling LLaVA-1.5-13B to even surpass the performance of GPT-4V.