WikiMixQA: A Multimodal Benchmark for Question Answering over Tables and Charts

TL;DR

WikiMixQA is a new benchmark for evaluating models on complex question answering tasks involving tables and charts from Wikipedia, highlighting current limitations in long-context multimodal reasoning.

Contribution

Introduces WikiMixQA, a challenging multimodal benchmark with 1,000 questions over Wikipedia content, emphasizing complex reasoning and long-context understanding.

Findings

Proprietary models reach ~70% accuracy with direct context

Performance drops significantly with long document retrieval

GPT-4-o exceeds 50% accuracy in retrieval scenarios

Abstract

Documents are fundamental to preserving and disseminating information, often incorporating complex layouts, tables, and charts that pose significant challenges for automatic document understanding (DU). While vision-language large models (VLLMs) have demonstrated improvements across various tasks, their effectiveness in processing long-context vision inputs remains unclear. This paper introduces WikiMixQA, a benchmark comprising 1,000 multiple-choice questions (MCQs) designed to evaluate cross-modal reasoning over tables and charts extracted from 4,000 Wikipedia pages spanning seven distinct topics. Unlike existing benchmarks, WikiMixQA emphasizes complex reasoning by requiring models to synthesize information from multiple modalities. We evaluate 12 state-of-the-art vision-language models, revealing that while proprietary models achieve ~70% accuracy when provided with direct context, their performance deteriorates significantly when retrieval from long documents is required. Among these, GPT-4-o is the only model exceeding 50% accuracy in this setting, whereas open-source models perform considerably worse, with a maximum accuracy of 27%. These findings underscore the challenges of long-context, multi-modal reasoning and establish WikiMixQA as a crucial benchmark for advancing document understanding research.