R-CoV: Region-Aware Chain-of-Verification for Alleviating Object Hallucinations in LVLMs

TL;DR

R-CoV is a post-hoc, region-aware verification method that reduces object hallucinations in large vision-language models by focusing on specific image regions and verifying their content without external detectors.

Contribution

The paper introduces R-CoV, a training-free, region-aware chain-of-verification approach that effectively mitigates object hallucinations in LVLMs.

Findings

R-CoV significantly reduces object hallucinations across multiple benchmarks.

The method can be integrated into various LVLMs without retraining.

It does not rely on external detection models, simplifying deployment.

Abstract

Large vision-language models (LVLMs) have demonstrated impressive performance in various multimodal understanding and reasoning tasks. However, they still struggle with object hallucinations, i.e., the claim of nonexistent objects in the visual input. To address this challenge, we propose Region-aware Chain-of-Verification (R-CoV), a visual chain-of-verification method to alleviate object hallucinations in LVLMs in a post-hoc manner. Motivated by how humans comprehend intricate visual information -- often focusing on specific image regions or details within a given sample -- we elicit such region-level processing from LVLMs themselves and use it as a chaining cue to detect and alleviate their own object hallucinations. Specifically, our R-CoV consists of six steps: initial response generation, entity extraction, coordinate generation, region description, verification execution, and final response generation. As a simple yet effective method, R-CoV can be seamlessly integrated into various LVLMs in a training-free manner and without relying on external detection models. Extensive experiments on several widely used hallucination benchmarks across multiple LVLMs demonstrate that R-CoV can significantly alleviate object hallucinations in LVLMs. Project page: https://github.com/Jiahao000/R-CoV.