Benchmarking Retrieval-Augmented Generation in Multi-Modal Contexts

TL;DR

This paper introduces M$^2$RAG, a benchmark for evaluating multi-modal retrieval-augmented generation models across four tasks, and proposes MM-RAIT, an instruction tuning method that significantly improves model performance in multi-modal contexts.

Contribution

The paper presents M$^2$RAG, a new benchmark for multi-modal RAG tasks, and introduces MM-RAIT, an instruction tuning technique that enhances multi-modal model responses.

Findings

MM-RAIT improves response quality significantly.

Outperforms MiniCPM-V 2.6 and Qwen2-VL by 34% and 33%.

Demonstrates effectiveness across four multi-modal tasks.

Abstract

With the rapid advancement of Multi-modal Large Language Models (MLLMs), their capability in understanding both images and text has greatly improved. However, their potential for leveraging multi-modal contextual information in Retrieval-Augmented Generation (RAG) remains largely underexplored. To address this gap, this paper introduces Multi-Modal Retrieval-Augmented Generation (M$^2$RAG), a benchmark designed to evaluate the effectiveness of Multi-modal Large Language Models in leveraging knowledge from multi-modal retrieval documents. The benchmark comprises four tasks: image captioning, multi-modal question answering, multi-modal fact verification, and image reranking. All tasks are set in an open-domain setting, requiring RAG models to retrieve query-relevant information from a multi-modal document collection and use it as contextual input for RAG modeling. To enhance the context utilization capabilities of MLLMs, we also introduce Multi-Modal Retrieval-Augmented Instruction Tuning (MM-RAIT), an instruction tuning method that optimizes MLLMs within multi-modal contexts. Our experiments demonstrate the effectiveness of MM-RAIT by significantly improving the quality of responses generated by different RAG models, outperforming MiniCPM-V 2.6 and Qwen2-VL with 34% and 33% gains, respectively. All data and code are available at https://github.com/NEUIR/M2RAG.