Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger

TL;DR

This paper introduces RCTS, a multimodal RAG framework that enhances large vision-language models by constructing a reasoning-enriched knowledge base and employing tree search re-ranking, leading to state-of-the-art VQA performance.

Contribution

It proposes a novel RAG framework with a reasoning-enriched knowledge base and Monte Carlo Tree Search re-ranking, improving reasoning and response consistency in LVLMs.

Findings

Achieves state-of-the-art results on multiple VQA datasets.

Outperforms In-Context Learning and Vanilla-RAG methods.

Enhances reasoning and response consistency in LVLMs.

Abstract

Recent advancements in Large Vision Language Models (LVLMs) have significantly improved performance in Visual Question Answering (VQA) tasks through multimodal Retrieval-Augmented Generation (RAG). However, existing methods still face challenges, such as the scarcity of knowledge with reasoning examples and erratic responses from retrieved knowledge. To address these issues, in this study, we propose a multimodal RAG framework, termed RCTS, which enhances LVLMs by constructing a Reasoning Context-enriched knowledge base and a Tree Search re-ranking method. Specifically, we introduce a self-consistent evaluation mechanism to enrich the knowledge base with intrinsic reasoning patterns. We further propose a Monte Carlo Tree Search with Heuristic Rewards (MCTS-HR) to prioritize the most relevant examples. This ensures that LVLMs can leverage high-quality contextual reasoning for better and more consistent responses. Extensive experiments demonstrate that our framework achieves state-of-the-art performance on multiple VQA datasets, significantly outperforming In-Context Learning (ICL) and Vanilla-RAG methods. It highlights the effectiveness of our knowledge base and re-ranking method in improving LVLMs. Our code is available at https://github.com/yannqi/RCTS-RAG.