Retrieval-augmented Multi-modal Chain-of-Thoughts Reasoning for Large Language Models

TL;DR

This paper introduces a retrieval-based multi-modal Chain-of-Thought approach that dynamically selects diverse demonstration examples, significantly improving reasoning performance of large language and multimodal models on benchmark datasets.

Contribution

It proposes a novel retrieval and stratified sampling method for selecting diverse, cross-modal, and intra-modal demonstration examples to enhance multi-modal reasoning in LLMs.

Findings

Improves GPT-4 performance by 6% on ScienceQA

Enhances GPT-4V accuracy by 2.7% on two datasets

Significantly boosts multi-modal reasoning capabilities of LLMs and LMMs

Abstract

The advancement of Large Language Models (LLMs) has brought substantial attention to the Chain of Thought (CoT) approach, primarily due to its ability to enhance the capability of LLMs on complex reasoning tasks. Moreover, the significance of CoT approaches extends to the application of LLMs for multi-modal tasks. However, the selection of optimal CoT demonstration examples in multi-modal reasoning remains less explored for LLMs due to the inherent complexity of multi-modal examples. In this paper, we introduce a novel approach that addresses this challenge by using retrieval mechanisms to dynamically and automatically select demonstration examples based on cross-modal and intra-modal similarities. Furthermore, we employ a Stratified Sampling method of categorising demonstration examples into groups based on their types and then retrieving examples from different groups respectively to promote the diversity of demonstration examples. Through a series of experiments on two popular benchmark datasets: ScienceQA and MathVista, we demonstrate that our approach significantly improves the performance of GPT-4 by 6% on ScienceQA and 12.9% on MathVista, and enhances the performance of GPT-4V on two datasets by 2.7%, substantially improving the performance of the most advanced LLMs and LMMs for complex multi-modal reasoning tasks.