AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding

TL;DR

AlignVLM introduces a novel method for aligning visual features with language embeddings by mapping them to a weighted average of LLM text embeddings, improving multimodal document understanding especially in low-resource scenarios.

Contribution

The paper presents AlignVLM, a new alignment technique that leverages linguistic priors in LLMs to better map visual features, outperforming existing methods in document understanding tasks.

Findings

Achieves state-of-the-art performance on document understanding tasks.

More effective in low-resource settings compared to prior methods.

Demonstrates robustness to noise and efficiency in alignment.

Abstract

Aligning visual features with language embeddings is a key challenge in vision-language models (VLMs). The performance of such models hinges on having a good connector that maps visual features generated by a vision encoder to a shared embedding space with the LLM while preserving semantic similarity. Existing connectors, such as multilayer perceptrons (MLPs), lack inductive bias to constrain visual features within the linguistic structure of the LLM's embedding space, making them data-hungry and prone to cross-modal misalignment. In this work, we propose a novel vision-text alignment method, AlignVLM, that maps visual features to a weighted average of LLM text embeddings. Our approach leverages the linguistic priors encoded by the LLM to ensure that visual features are mapped to regions of the space that the LLM can effectively interpret. AlignVLM is particularly effective for document understanding tasks, where visual and textual modalities are highly correlated. Our extensive experiments show that AlignVLM achieves state-of-the-art performance compared to prior alignment methods, with larger gains on document understanding tasks and under low-resource setups. We provide further analysis demonstrating its efficiency and robustness to noise.