CREM: Compression-Driven Representation Enhancement for Multimodal Retrieval and Comprehension

TL;DR

CREM is a unified model that enhances multimodal representations for retrieval tasks while maintaining generative capabilities, using compression-based prompts and training strategies to align cross-modal semantics.

Contribution

The paper introduces CREM, a novel framework that combines compression-driven prompts and training to improve retrieval and comprehension in multimodal large language models.

Findings

Achieves state-of-the-art retrieval performance on MMEB.

Maintains strong generative performance on multiple benchmarks.

Generative supervision enhances representation quality.

Abstract

Multimodal Large Language Models (MLLMs) have shown remarkable success in comprehension tasks such as visual description and visual question answering. However, their direct application to embedding-based tasks like retrieval remains challenging due to the discrepancy between output formats and optimization objectives. Previous approaches often employ contrastive fine-tuning to adapt MLLMs for retrieval, but at the cost of losing their generative capabilities. We argue that both generative and embedding tasks fundamentally rely on shared cognitive mechanisms, specifically cross-modal representation alignment and contextual comprehension. To this end, we propose CREM (Compression-driven Representation Enhanced Model), with a unified framework that enhances multimodal representations for retrieval while preserving generative ability. Specifically, we introduce a compression-based prompt design with learnable chorus tokens to aggregate multimodal semantics and a compression-driven training strategy that integrates contrastive and generative objectives through compression-aware attention. Extensive experiments demonstrate that CREM achieves state-of-the-art retrieval performance on MMEB while maintaining strong generative performance on multiple comprehension benchmarks. Our findings highlight that generative supervision can further improve the representational quality of MLLMs under the proposed compression-driven paradigm.