MCA-LLaVA: Manhattan Causal Attention for Reducing Hallucination in Large Vision-Language Models

TL;DR

This paper introduces MCA-LLaVA, a novel Manhattan distance-based positional encoding method that reduces hallucinations in large vision-language models by improving multimodal alignment.

Contribution

It proposes a two-dimensional spatial decay mechanism extending RoPE to mitigate image alignment bias in LVLMs.

Findings

MCA-LLaVA effectively reduces hallucinations across benchmarks.

The method improves multimodal feature alignment.

Experimental results demonstrate its generality and effectiveness.

Abstract

Hallucinations pose a significant challenge in Large Vision Language Models (LVLMs), with misalignment between multimodal features identified as a key contributing factor. This paper reveals the negative impact of the long-term decay in Rotary Position Encoding (RoPE), used for positional modeling in LVLMs, on multimodal alignment. Concretely, under long-term decay, instruction tokens exhibit uneven perception of image tokens located at different positions within the two-dimensional space: prioritizing image tokens from the bottom-right region since in the one-dimensional sequence, these tokens are positionally closer to the instruction tokens. This biased perception leads to insufficient image-instruction interaction and suboptimal multimodal alignment. We refer to this phenomenon as image alignment bias. To enhance instruction's perception of image tokens at different spatial locations, we propose MCA-LLaVA, based on Manhattan distance, which extends the long-term decay to a two-dimensional, multi-directional spatial decay. MCA-LLaVA integrates the one-dimensional sequence order and two-dimensional spatial position of image tokens for positional modeling, mitigating hallucinations by alleviating image alignment bias. Experimental results of MCA-LLaVA across various hallucination and general benchmarks demonstrate its effectiveness and generality. The code can be accessed in https://github.com/ErikZ719/MCA-LLaVA.