Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation

TL;DR

This survey comprehensively reviews Multimodal Retrieval-Augmented Generation, highlighting datasets, methodologies, challenges, and future directions for integrating multiple modalities to improve AI's factual accuracy and reasoning capabilities.

Contribution

It provides a structured analysis of Multimodal RAG systems, covering recent advancements, challenges, and open problems, serving as a foundational resource for future research.

Findings

Analyzes datasets, benchmarks, and evaluation metrics for Multimodal RAG.

Summarizes recent methodologies and innovations in retrieval and fusion techniques.

Identifies open challenges and future research directions in multimodal reasoning.

Abstract

Large Language Models (LLMs) suffer from hallucinations and outdated knowledge due to their reliance on static training data. Retrieval-Augmented Generation (RAG) mitigates these issues by integrating external dynamic information for improved factual grounding. With advances in multimodal learning, Multimodal RAG extends this approach by incorporating multiple modalities such as text, images, audio, and video to enhance the generated outputs. However, cross-modal alignment and reasoning introduce unique challenges beyond those in unimodal RAG. This survey offers a structured and comprehensive analysis of Multimodal RAG systems, covering datasets, benchmarks, metrics, evaluation, methodologies, and innovations in retrieval, fusion, augmentation, and generation. We review training strategies, robustness enhancements, loss functions, and agent-based approaches, while also exploring the diverse Multimodal RAG scenarios. In addition, we outline open challenges and future directions to guide research in this evolving field. This survey lays the foundation for developing more capable and reliable AI systems that effectively leverage multimodal dynamic external knowledge bases. All resources are publicly available at https://github.com/llm-lab-org/Multimodal-RAG-Survey.