SEPS: Semantic-enhanced Patch Slimming Framework for fine-grained cross-modal alignment

TL;DR

SEPS is a novel framework that enhances fine-grained cross-modal alignment by systematically reducing patch redundancy and ambiguity through semantic integration and relevance-aware patch selection, significantly improving retrieval performance.

Contribution

The paper introduces SEPS, a two-stage semantic-enhanced patch slimming framework that effectively addresses patch redundancy and ambiguity in cross-modal alignment tasks.

Findings

SEPS outperforms existing methods by 23-86% in rSum on Flickr30K and MS-COCO datasets.

The framework improves text-to-image retrieval accuracy across various model architectures.

Experimental results demonstrate the effectiveness of semantic integration and relevance-aware patch selection.

Abstract

Fine-grained cross-modal alignment aims to establish precise local correspondences between vision and language, forming a cornerstone for visual question answering and related multimodal applications. Current approaches face challenges in addressing patch redundancy and ambiguity, which arise from the inherent information density disparities across modalities. Recently, Multimodal Large Language Models (MLLMs) have emerged as promising solutions to bridge this gap through their robust semantic generation capabilities. However, the dense textual outputs from MLLMs may introduce conflicts with the original sparse captions. Furthermore, accurately quantifying semantic relevance between rich visual patches and concise textual descriptions remains a core challenge. To overcome these limitations, we introduce the Semantic-Enhanced Patch Slimming (SEPS) framework, which systematically addresses patch redundancy and ambiguity. Our approach employs a two-stage mechanism to integrate unified semantics from both dense and sparse texts, enabling the identification of salient visual patches. Additionally, it leverages relevance-aware selection with mean value computation to highlight crucial patch-word correspondences, thereby improving cross-modal similarity assessment. Comprehensive experiments on Flickr30K and MS-COCO datasets validate that SEPS achieves superior performance, surpassing existing approaches by 23\%-86\% in rSum across diverse model architectures, with notable enhancements in text-to-image retrieval scenarios. Our implementation is available at https://github.com/Sweet4tars/seps.git.