MMAPG: A Training-Free Framework for Multimodal Multi-hop Question Answering via Adaptive Planning Graphs

TL;DR

This paper introduces MMAPG, a training-free framework for multimodal multi-hop question answering that uses adaptive planning graphs to dynamically explore reasoning paths, reducing training costs and improving performance.

Contribution

The paper presents a novel training-free approach with adaptive planning graphs and modality-specific retrieval strategies for multimodal QA.

Findings

Matches or outperforms training-based models on MultimodalQA and WebQA datasets.

Reduces computational costs by eliminating the need for task-specific training.

Enables flexible and dynamic reasoning path exploration in multimodal QA.

Abstract

Multimodal Multi-hop question answering requires integrating information from diverse sources, such as images and texts, to derive answers. Existing methods typically rely on sequential retrieval and reasoning, where each step builds on the previous output. However, this single-path paradigm makes them vulnerable to errors due to misleading intermediate steps. Moreover, developing multimodal models can be computationally expensive, often requiring extensive training. To address these limitations, we propose a training-free framework guided by an Adaptive Planning Graph, which consists of planning, retrieval and reasoning modules. The planning module analyzes the current state of the Adaptive Planning Graph, determines the next action and where to expand the graph, which enables dynamic and flexible exploration of reasoning paths. To handle retrieval of text to unspecified target modalities, we devise modality-specific strategies that dynamically adapt to distinct data types. Our approach preserves the characteristics of multimodal information without costly task-specific training, enabling seamless integration with up-to-date models. Finally, the experiments on MultimodalQA and WebQA show that our approach matches or outperforms existing models that rely on training.