CT2C-QA: Multimodal Question Answering over Chinese Text, Table and Chart

TL;DR

This paper introduces CT2C-QA, a comprehensive Chinese multimodal QA dataset combining text, tables, and charts, along with a multi-agent system called AED for reasoning, highlighting current models' limitations in handling such complex data.

Contribution

The paper presents the first Chinese multimodal QA dataset with text, tables, and charts, and proposes a multi-agent reasoning system, AED, to improve analysis and decision-making.

Findings

Current models, including GPT-4, underperform on the dataset.

The AED system outperforms existing models in multimodal reasoning tasks.

The dataset effectively tests models' ability to analyze diverse data modalities.

Abstract

Multimodal Question Answering (MMQA) is crucial as it enables comprehensive understanding and accurate responses by integrating insights from diverse data representations such as tables, charts, and text. Most existing researches in MMQA only focus on two modalities such as image-text QA, table-text QA and chart-text QA, and there remains a notable scarcity in studies that investigate the joint analysis of text, tables, and charts. In this paper, we present C$\text{T}^2$C-QA, a pioneering Chinese reasoning-based QA dataset that includes an extensive collection of text, tables, and charts, meticulously compiled from 200 selectively sourced webpages. Our dataset simulates real webpages and serves as a great test for the capability of the model to analyze and reason with multimodal data, because the answer to a question could appear in various modalities, or even potentially not exist at all. Additionally, we present AED (\textbf{A}llocating, \textbf{E}xpert and \textbf{D}esicion), a multi-agent system implemented through collaborative deployment, information interaction, and collective decision-making among different agents. Specifically, the Assignment Agent is in charge of selecting and activating expert agents, including those proficient in text, tables, and charts. The Decision Agent bears the responsibility of delivering the final verdict, drawing upon the analytical insights provided by these expert agents. We execute a comprehensive analysis, comparing AED with various state-of-the-art models in MMQA, including GPT-4. The experimental outcomes demonstrate that current methodologies, including GPT-4, are yet to meet the benchmarks set by our dataset.