Mema: Memory-Augmented Adapter for Enhanced Vision-Language Understanding

TL;DR

Mema is a memory-augmented adapter that enhances vision-language understanding by hierarchically integrating visual cues across layers, improving multimodal reasoning with minimal additional training.

Contribution

Introduces a lightweight, plug-and-play memory module that captures hierarchical visual features and improves multimodal model performance without altering the backbone.

Findings

Mema improves performance across multiple benchmarks.

The memory mechanism effectively preserves fine-grained visual cues.

The approach requires minimal additional training parameters.

Abstract

Multimodal Large Language Models (MLLMs) have achieved remarkable performance by aligning pretrained visual representations with the linguistic knowledge embedded in Large Language Models (LLMs). However, existing approaches typically rely on final-layer visual features or learnable multi-layer fusion, which often fail to sufficiently exploit hierarchical visual cues without explicit cross-layer interaction design. In this work, we propose a Memory-Augmented Adapter (Mema) within the vision encoder. Specifically, Mema maintains a stateful memory that accumulates hierarchical visual representations across layers, with its evolution conditioned on both query embeddings and step-wise visual features. A portion of this memory is selectively injected into token representations via a feedback mechanism, thereby mitigating the attenuation of fine-grained visual cues from shallow layers. Designed as a lightweight and plug-and-play module, Mema integrates seamlessly into pretrained vision encoders without modifying the vanilla backbone architecture. Only a minimal set of additional parameters requires training, enabling adaptive visual feature refinement while reducing training overhead. Extensive experiments across multiple benchmarks demonstrate that Mema consistently improves performance, validating its effectiveness in complex multimodal reasoning tasks. The code have been released at https://github.com/Sisiliu312/Mema.