Mitigating Object Hallucination via Robust Local Perception Search

TL;DR

This paper introduces Local Perception Search (LPS), a training-free decoding method that reduces hallucinations in multimodal large language models by leveraging local visual priors, especially effective in noisy image scenarios.

Contribution

The paper proposes LPS, a simple, training-free decoding technique that effectively mitigates hallucinations in multimodal models by utilizing local visual priors during inference.

Findings

LPS significantly reduces hallucinations on benchmark datasets.

LPS performs well in noisy image conditions.

LPS is compatible with various multimodal models.

Abstract

Recent advancements in Multimodal Large Language Models (MLLMs) have enabled them to effectively integrate vision and language, addressing a variety of downstream tasks. However, despite their significant success, these models still exhibit hallucination phenomena, where the outputs appear plausible but do not align with the content of the images. To mitigate this issue, we introduce Local Perception Search (LPS), a decoding method during inference that is both simple and training-free, yet effectively suppresses hallucinations. This method leverages local visual prior information as a value function to correct the decoding process. Additionally, we observe that the impact of the local visual prior on model performance is more pronounced in scenarios with high levels of image noise. Notably, LPS is a plug-and-play approach that is compatible with various models. Extensive experiments on widely used hallucination benchmarks and noisy data demonstrate that LPS significantly reduces the incidence of hallucinations compared to the baseline, showing exceptional performance, particularly in noisy settings.