Multimodal Multi-Hop Question Answering Through a Conversation Between Tools and Efficiently Finetuned Large Language Models

TL;DR

This paper introduces a divide-and-conquer approach where large language models interact with tools to answer complex multimodal multi-hop questions, significantly improving performance over existing methods.

Contribution

The authors propose a novel tool-interacting divide-and-conquer strategy and an efficient finetuning method for large language models to handle complex multimodal multi-hop questions.

Findings

Substantial performance improvements over state-of-the-art methods.

Effective use of a generated dataset for finetuning LLMs.

Demonstrated generality across multiple complex question datasets.

Abstract

We employ a tool-interacting divide-and-conquer strategy enabling large language models (LLMs) to answer complex multimodal multi-hop questions. In particular, we harness the power of large language models to divide a given multimodal multi-hop question into unimodal single-hop sub-questions to be answered by the appropriate tool from a predefined set of tools. After all corresponding tools provide the LLM with their answers, the LLM generates the next relevant unimodal single-hop question. To increase the reasoning ability of LLMs, we prompt chatGPT to generate a tool-interacting divide-and-conquer dataset. This dataset is then used to efficiently finetune the corresponding LLM. To assess the effectiveness of this approach, we conduct an evaluation on two recently introduced complex question-answering datasets. The experimental analysis demonstrate substantial improvements over existing state-of-the-art solutions, indicating the efficacy and generality of our strategy