Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD)

TL;DR

This paper introduces Language Contrastive Decoding (LCD), a novel method to reduce object hallucinations in Large Vision-Language Models by adjusting output confidence levels, leading to improved accuracy and caption quality.

Contribution

The study proposes LCD, a new decoding algorithm that effectively mitigates hallucinations in LVLMs without retraining or complex post-processing, applicable across different models.

Findings

Up to 4% improvement in POPE F1 scores

Up to 36% reduction in CHAIR scores

Enhanced captioning quality scores

Abstract

Large Vision-Language Models (LVLMs) are an extension of Large Language Models (LLMs) that facilitate processing both image and text inputs, expanding AI capabilities. However, LVLMs struggle with object hallucinations due to their reliance on text cues and learned object co-occurrence biases. While most research quantifies these hallucinations, mitigation strategies are still lacking. Our study introduces a Language Contrastive Decoding (LCD) algorithm that adjusts LVLM outputs based on LLM distribution confidence levels, effectively reducing object hallucinations. We demonstrate the advantages of LCD in leading LVLMs, showing up to %4 improvement in POPE F1 scores and up to %36 reduction in CHAIR scores on the COCO validation set, while also improving captioning quality scores. Our method effectively improves LVLMs without needing complex post-processing or retraining, and is easily applicable to different models. Our findings highlight the potential of further exploration of LVLM-specific decoding algorithms.