HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding

TL;DR

HALC is a decoding algorithm that reduces object hallucinations in large vision-language models by combining local auto-focal grounding and global beam search, improving accuracy without retraining.

Contribution

HALC introduces a novel plug-and-play decoding method that mitigates object hallucinations in LVLMs by leveraging fine-grained visual information and adaptive search strategies.

Findings

HALC significantly reduces object hallucinations across multiple benchmarks.

It maintains high text generation quality while improving factual accuracy.

Outperforms existing methods in object hallucination mitigation.

Abstract

While large vision-language models (LVLMs) have demonstrated impressive capabilities in interpreting multi-modal contexts, they invariably suffer from object hallucinations (OH). We introduce HALC, a novel decoding algorithm designed to mitigate OH in LVLMs. HALC leverages distinct fine-grained optimal visual information in vision-language tasks and operates on both local and global contexts simultaneously. Specifically, HALC integrates a robust auto-focal grounding mechanism (locally) to correct hallucinated tokens on the fly, and a specialized beam search algorithm (globally) to significantly reduce OH while preserving text generation quality. Additionally, HALC can be integrated into any LVLMs as a plug-and-play module without extra training. Extensive experimental studies demonstrate the effectiveness of HALC in reducing OH, outperforming state-of-the-arts across four benchmarks.