Energy-Guided Decoding for Object Hallucination Mitigation

TL;DR

This paper introduces an energy-based decoding method for large vision-language models that effectively reduces object hallucination bias and improves accuracy in visual question answering tasks.

Contribution

The paper proposes a simple energy-based decoding approach that dynamically selects hidden states to mitigate hallucination bias in VLMs, outperforming existing methods.

Findings

Reduces yes-ratio bias by 8.81% on average

Improves accuracy by 4.82% over greedy decoding

Enhances performance across three VQA benchmarks

Abstract

Mitigating object hallucination in large vision-language models (LVLMs) is critical to their safe deployment. Existing methods either are restricted to specific decoding methods, or demand sophisticated modifications to visual inputs, or rely on knowledge from external models. In this work, we first reveal the phenomenon that VLMs exhibit significant imbalance in the ``Yes'' ratio ( \ie, the fraction of ``Yes'' answers among the total number of questions) across three different visual question answering (VQA) datasets. Furthermore, we propose an energy-based decoding method, which dynamically selects the hidden states from the layer with minimal energy score. It is simple yet effective in reducing the bias for the yes ratio while boosting performance across three benchmarks (POPE, MME, and MMVP). Our method consistently improves accuracy and F1 score on three VQA datasets across three commonly used VLMs over several baseline methods. The average accuracy improvement is 4.82% compared to greedy decoding. Moreover, the average yes-ratio gap reduction is 8.81%, meaning the proposed method is less biased as shown in Figure 1.