A General Retrieval-Augmented Generation Framework for Multimodal Case-Based Reasoning Applications

TL;DR

This paper introduces MCBR-RAG, a versatile framework that enables retrieval-augmented generation for multimodal case-based reasoning by converting non-text components into text for improved retrieval and context enrichment.

Contribution

It proposes a novel framework that extends retrieval-augmented generation to multimodal cases by converting components into text, enhancing retrieval and reasoning in CBR applications.

Findings

MCBR-RAG improves generation quality over baseline LLMs.

Effective in both simple and complex multimodal scenarios.

Enhances context utilization in multimodal case-based reasoning.

Abstract

Case-based reasoning (CBR) is an experience-based approach to problem solving, where a repository of solved cases is adapted to solve new cases. Recent research shows that Large Language Models (LLMs) with Retrieval-Augmented Generation (RAG) can support the Retrieve and Reuse stages of the CBR pipeline by retrieving similar cases and using them as additional context to an LLM query. Most studies have focused on text-only applications, however, in many real-world problems the components of a case are multimodal. In this paper we present MCBR-RAG, a general RAG framework for multimodal CBR applications. The MCBR-RAG framework converts non-text case components into text-based representations, allowing it to: 1) learn application-specific latent representations that can be indexed for retrieval, and 2) enrich the query provided to the LLM by incorporating all case components for better context. We demonstrate MCBR-RAG's effectiveness through experiments conducted on a simplified Math-24 application and a more complex Backgammon application. Our empirical results show that MCBR-RAG improves generation quality compared to a baseline LLM with no contextual information provided.