See Different, Think Better: Visual Variations Mitigating Hallucinations in LVLMs

TL;DR

This paper introduces ViHallu, a visual-centric framework that uses visual variations and instructions to improve fine-grained visual understanding in LVLMs, significantly reducing hallucinations and enhancing visual-semantic alignment.

Contribution

The paper proposes a novel visual variation generation method and visual instruction construction to mitigate hallucinations in LVLMs, focusing on visual-semantic alignment improvements.

Findings

ViHallu reduces hallucination tendencies in LVLMs.

Enhanced fine-grained visual understanding demonstrated on multiple benchmarks.

Release of ViHallu-Instruction dataset for hallucination mitigation.

Abstract

Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities in visual understanding and multimodal reasoning. However, LVLMs frequently exhibit hallucination phenomena, manifesting as the generated textual responses that demonstrate inconsistencies with the provided visual content. Existing hallucination mitigation methods are predominantly text-centric, the challenges of visual-semantic alignment significantly limit their effectiveness, especially when confronted with fine-grained visual understanding scenarios. To this end, this paper presents ViHallu, a Vision-Centric Hallucination mitigation framework that enhances visual-semantic alignment through Visual Variation Image Generation and Visual Instruction Construction. ViHallu introduces visual variation images with controllable visual alterations while maintaining the overall image structure. These images, combined with carefully constructed visual instructions, enable LVLMs to better understand fine-grained visual content through fine-tuning, allowing models to more precisely capture the correspondence between visual content and text, thereby enhancing visual-semantic alignment. Extensive experiments on multiple benchmarks show that ViHallu effectively enhances models' fine-grained visual understanding while significantly reducing hallucination tendencies. Furthermore, we release ViHallu-Instruction, a visual instruction dataset specifically designed for hallucination mitigation and visual-semantic alignment. Code is available at https://github.com/oliviadzy/ViHallu.