Adaptive Global and Fine-Grained Perceptual Fusion for MLLM Embeddings Compatible with Hard Negative Amplification

TL;DR

This paper introduces AGFF-Embed, a novel fusion method for MLLM embeddings that adaptively combines global and fine-grained semantic information, enhancing multimodal understanding and hard negative mining.

Contribution

It proposes a new adaptive fusion mechanism for MLLM embeddings and integrates gradient amplification for improved hard negative mining, advancing multimodal embedding performance.

Findings

Achieves state-of-the-art results on MMEB and MMVP-VLM benchmarks.

Effectively combines global and fine-grained semantic information.

Enhances hard negative sampling without dataset modifications.

Abstract

Multimodal embeddings serve as a bridge for aligning vision and language, with the two primary implementations -- CLIP-based and MLLM-based embedding models -- both limited to capturing only global semantic information. Although numerous studies have focused on fine-grained understanding, we observe that complex scenarios currently targeted by MLLM embeddings often involve a hybrid perceptual pattern of both global and fine-grained elements, thus necessitating a compatible fusion mechanism. In this paper, we propose Adaptive Global and Fine-grained perceptual Fusion for MLLM Embeddings (AGFF-Embed), a method that prompts the MLLM to generate multiple embeddings focusing on different dimensions of semantic information, which are then adaptively and smoothly aggregated. Furthermore, we adapt AGFF-Embed with the Explicit Gradient Amplification (EGA) technique to achieve in-batch hard negatives enhancement without requiring fine-grained editing of the dataset. Evaluation on the MMEB and MMVP-VLM benchmarks shows that AGFF-Embed comprehensively achieves state-of-the-art performance in both general and fine-grained understanding compared to other multimodal embedding models.