SEA: Supervised Embedding Alignment for Token-Level Visual-Textual Integration in MLLMs

TL;DR

This paper introduces SEA, a token-level supervised embedding alignment method that enhances visual-textual integration in multimodal large language models, especially benefiting smaller models by significantly improving their cross-modal understanding.

Contribution

SEA provides a novel token-level supervision approach for better modality alignment, outperforming traditional adapter-based methods with minimal additional computational cost.

Findings

SEA improves performance across various model sizes.

Smaller models see an average gain of 7.61%.

SEA enhances cross-modal understanding significantly.

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities by integrating visual and textual inputs, yet modality alignment remains one of the most challenging aspects. Current MLLMs typically rely on simple adapter architectures and pretraining approaches to bridge vision encoders with large language models (LLM), guided by image-level supervision. We identify this paradigm often leads to suboptimal alignment between modalities, significantly constraining the LLM's ability to properly interpret and reason with visual features particularly for smaller language models. This limitation degrades overall performance-particularly for smaller language models where capacity constraints are more pronounced and adaptation capabilities are limited. To address this fundamental limitation, we propose Supervised Embedding Alignment (SEA), a token-level supervision alignment method that enables more precise visual-text alignment during pretraining. SEA introduces minimal computational overhead while preserving language capabilities and substantially improving cross-modal understanding. Our comprehensive analyses reveal critical insights into the adapter's role in multimodal integration, and extensive experiments demonstrate that SEA consistently improves performance across various model sizes, with smaller models benefiting the most (average performance gain of 7.61% for Gemma-2B). This work establishes a foundation for developing more effective alignment strategies for future multimodal systems.