MAGE: Multimodal Alignment and Generation Enhancement via Bridging Visual and Semantic Spaces

TL;DR

MAGE is a novel multimodal framework that improves visual and semantic alignment between vision and language models, enhancing generation capabilities and performance across multiple benchmarks.

Contribution

MAGE introduces an innovative alignment mechanism with IAN and a training strategy to bridge semantic gaps, expanding multimodal model capabilities.

Findings

Significantly outperforms similar models on MME, MMBench, and SEED benchmarks.

Achieves better semantic and dimensional alignment between visual and textual data.

Enhances 'Any-to-Any' multimodal generation capabilities.

Abstract

In the latest advancements in multimodal learning, effectively addressing the spatial and semantic losses of visual data after encoding remains a critical challenge. This is because the performance of large multimodal models is positively correlated with the coupling between visual encoders and large language models. Existing approaches often face issues such as vector gaps or semantic disparities, resulting in information loss during the propagation process. To address these issues, we propose MAGE (Multimodal Alignment and Generation Enhancement), a novel framework that bridges the semantic spaces of vision and text through an innovative alignment mechanism. By introducing the Intelligent Alignment Network (IAN), MAGE achieves dimensional and semantic alignment. To reduce the gap between synonymous heterogeneous data, we employ a training strategy that combines cross-entropy and mean squared error, significantly enhancing the alignment effect. Moreover, to enhance MAGE's "Any-to-Any" capability, we developed a fine-tuning dataset for multimodal tool-calling instructions to expand the model's output capability boundaries. Finally, our proposed multimodal large model architecture, MAGE, achieved significantly better performance compared to similar works across various evaluation benchmarks, including MME, MMBench, and SEED. Complete code and appendix are available at: https://github.com/GTCOM-NLP/MAGE.