MoqaGPT : Zero-Shot Multi-modal Open-domain Question Answering with Large Language Model

TL;DR

MoqaGPT is a flexible, zero-shot framework that enables large language models to perform multi-modal open-domain question answering by retrieving and fusing information from diverse modalities without complex ranking.

Contribution

It introduces a divide-and-conquer approach allowing LLMs to handle multiple modalities seamlessly in a zero-shot setting, improving performance on key datasets.

Findings

Significant performance improvements on MMCoQA dataset (+37.91 F1, +34.07 EM)

Outperforms zero-shot baselines on MultiModalQA (+9.5 F1, +10.1 EM)

Closes gap with supervised methods in multi-modal QA

Abstract

Multi-modal open-domain question answering typically requires evidence retrieval from databases across diverse modalities, such as images, tables, passages, etc. Even Large Language Models (LLMs) like GPT-4 fall short in this task. To enable LLMs to tackle the task in a zero-shot manner, we introduce MoqaGPT, a straightforward and flexible framework. Using a divide-and-conquer strategy that bypasses intricate multi-modality ranking, our framework can accommodate new modalities and seamlessly transition to new models for the task. Built upon LLMs, MoqaGPT retrieves and extracts answers from each modality separately, then fuses this multi-modal information using LLMs to produce a final answer. Our methodology boosts performance on the MMCoQA dataset, improving F1 by +37.91 points and EM by +34.07 points over the supervised baseline. On the MultiModalQA dataset, MoqaGPT surpasses the zero-shot baseline, improving F1 by 9.5 points and EM by 10.1 points, and significantly closes the gap with supervised methods. Our codebase is available at https://github.com/lezhang7/MOQAGPT.