Optimizing Vision-Language Consistency via Cross-Layer Regional Attention Alignment

TL;DR

This paper introduces a novel attention alignment method for vision-language models that enhances cross-modal consistency and interpretability, leading to state-of-the-art results with minimal additional parameters.

Contribution

The paper proposes CCRA, a new framework combining LPWCA and PAI to improve regional and semantic attention alignment in vision-language models.

Findings

Achieves state-of-the-art performance on ten benchmarks.

Enhances interpretability with regionally focused attention.

Requires only 3.55M additional parameters.

Abstract

Vision Language Models (VLMs) face challenges in effectively coordinating diverse attention mechanisms for cross-modal embedding learning, leading to mismatched attention and suboptimal performance. We propose Consistent Cross-layer Regional Alignment (CCRA), which introduces Layer-Patch-wise Cross Attention (LPWCA) to capture fine-grained regional-semantic correlations by jointly weighting patch and layer-wise embedding, and Progressive Attention Integration (PAI) that systematically coordinates LPWCA, layer-wise, and patch-wise attention mechanisms in sequence. This progressive design ensures consistency from semantic to regional levels while preventing attention drift and maximizing individual attention benefits. Experimental results on ten diverse vision-language benchmarks demonstrate that our CCRA-enhanced LLaVA-v1.5-7B model achieves state-of-the-art performance, outperforming all baseline methods with only 3.55M additional parameters, while providing enhanced interpretability through more regionally focused and semantically aligned attention patterns.