Cross-modal RAG: Sub-dimensional Text-to-Image Retrieval-Augmented Generation

TL;DR

Cross-modal RAG introduces a sub-dimensional retrieval and generation framework that decomposes queries and images into components, enabling more precise retrieval and synthesis in complex text-to-image tasks.

Contribution

It presents a novel sub-dimensional decomposition approach with hybrid retrieval and subquery-aware generation, improving over existing RAG methods for complex image synthesis.

Findings

Outperforms baselines in retrieval accuracy on multiple datasets.

Enhances generation quality with subquery-aware visual conditioning.

Maintains high efficiency in retrieval and generation processes.

Abstract

Text-to-image generation increasingly demands access to domain-specific, fine-grained, and rapidly evolving knowledge that pretrained models cannot fully capture, necessitating the integration of retrieval methods. Existing Retrieval-Augmented Generation (RAG) methods attempt to address this by retrieving globally relevant images, but they fail when no single image contains all desired elements from a complex user query. We propose Cross-modal RAG, a novel framework that decomposes both queries and images into sub-dimensional components, enabling subquery-aware retrieval and generation. Our method introduces a hybrid retrieval strategy - combining a sub-dimensional sparse retriever with a dense retriever - to identify a Pareto-optimal set of images, each contributing complementary aspects of the query. During generation, a multimodal large language model is guided to selectively condition on relevant visual features aligned to specific subqueries, ensuring subquery-aware image synthesis. Extensive experiments on MS-COCO, Flickr30K, WikiArt, CUB, and ImageNet-LT demonstrate that Cross-modal RAG significantly outperforms existing baselines in the retrieval and further contributes to generation quality, while maintaining high efficiency.