PUMA: Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learning

TL;DR

PUMA introduces a layer-pruning and modality-adaptive learning approach to make multimodal retrieval more efficient, reducing resource consumption while maintaining high performance in large language models.

Contribution

It presents a novel layer-pruning self-distillation method and a modality-adaptive contrastive loss for efficient and effective multimodal retrieval.

Findings

Significantly reduces model parameters and training costs.

Maintains strong retrieval performance with less resource usage.

Enhances learning by separating intra- and inter-modality negatives.

Abstract

As multimedia content expands, the demand for unified multimodal retrieval (UMR) in real-world applications increases. Recent work leverages multimodal large language models (MLLMs) to tackle this task. However, their large parameter size results in high training costs and low inference efficiency. To address this, we propose PUMA: a Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learning. Our approach improves UMR from both structural and learning perspectives. (1) Structurally, we propose Layer-Pruned Self-Distillation, which prunes MLLMs by keeping only shallow layers while distilling features from dropped deep layers as teacher signals. This reduces parameters and preserves representation capability. (2) On the learning side, we introduce Modality-Adaptive Contrastive Learning Loss (MAC-Loss), which separates in-batch negatives into harder intra-modality and easier inter-modality groups based on the target modality, assigning different temperature strategies to enhance learning efficiency. Experiments show our method significantly reduces resource usage while maintaining strong performance.