EMMA: Empowering Multi-modal Mamba with Structural and Hierarchical Alignment

TL;DR

EMMA enhances multi-modal Mamba models by introducing structural and hierarchical alignment techniques, significantly improving visual feature extraction, reducing hallucinations, and increasing inference speed across various benchmarks.

Contribution

The paper introduces pixel-wise alignment and multi-scale feature fusion modules to improve cross-modal alignment in Mamba-based multi-modal models, achieving faster inference and better performance.

Findings

Lower latency than other Mamba-based models

Nearly four times faster inference than transformer-based models

Improved accuracy and reduced hallucination in multi-modal tasks

Abstract

Mamba-based architectures have shown to be a promising new direction for deep learning models owing to their competitive performance and sub-quadratic deployment speed. However, current Mamba multi-modal large language models (MLLM) are insufficient in extracting visual features, leading to imbalanced cross-modal alignment between visual and textural latents, negatively impacting performance on multi-modal tasks. In this work, we propose Empowering Multi-modal Mamba with Structural and Hierarchical Alignment (EMMA), which enables the MLLM to extract fine-grained visual information. Specifically, we propose a pixel-wise alignment module to autoregressively optimize the learning and processing of spatial image-level features along with textual tokens, enabling structural alignment at the image level. In addition, to prevent the degradation of visual information during the cross-model alignment process, we propose a multi-scale feature fusion (MFF) module to combine multi-scale visual features from intermediate layers, enabling hierarchical alignment at the feature level. Extensive experiments are conducted across a variety of multi-modal benchmarks. Our model shows lower latency than other Mamba-based MLLMs and is nearly four times faster than transformer-based MLLMs of similar scale during inference. Due to better cross-modal alignment, our model exhibits lower degrees of hallucination and enhanced sensitivity to visual details, which manifests in superior performance across diverse multi-modal benchmarks. Code will be provided.