CART: A Generative Cross-Modal Retrieval Framework with Coarse-To-Fine Semantic Modeling

TL;DR

CART introduces a generative cross-modal retrieval framework that employs coarse-to-fine semantic modeling with discretized multimodal data, enhancing retrieval accuracy and efficiency.

Contribution

The paper presents a novel generative retrieval framework using coarse-to-fine semantic modeling and discretization techniques, reducing training costs and inference latency.

Findings

Achieves superior retrieval performance compared to traditional methods.

Demonstrates improved efficiency in large-scale cross-modal retrieval.

Validates effectiveness through extensive experiments.

Abstract

Cross-modal retrieval aims to search for instances, which are semantically related to the query through the interaction of different modal data. Traditional solutions utilize a single-tower or dual-tower framework to explicitly compute the score between queries and candidates, which is challenged by training cost and inference latency with large-scale data. Inspired by the remarkable performance and efficiency of generative models, we propose a generative cross-modal retrieval framework (CART) based on coarse-to-fine semantic modeling, which assigns identifiers to each candidate and treats the generating identifier as the retrieval target. Specifically, we explore an effective coarse-to-fine scheme, combining K-Means and RQ-VAE to discretize multimodal data into token sequences that support autoregressive generation. Further, considering the lack of explicit interaction between queries and candidates, we propose a feature fusion strategy to align their semantics. Extensive experiments demonstrate the effectiveness of the strategies in the CART, achieving excellent results in both retrieval performance and efficiency.