TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergy

TL;DR

TabPedia introduces a unified vision-language model with a concept synergy mechanism that integrates multiple visual table understanding tasks, enhancing comprehension and perception through large language models, and establishes a new comprehensive table VQA benchmark.

Contribution

The paper proposes a novel large vision-language model, TabPedia, with a concept synergy mechanism that unifies diverse VTU tasks and introduces a new benchmark, ComTQA, for real-world evaluation.

Findings

TabPedia achieves superior performance on various VTU benchmarks.

The concept synergy mechanism effectively integrates perception and comprehension tasks.

ComTQA provides a comprehensive dataset for real-world table VQA evaluation.

Abstract

Tables contain factual and quantitative data accompanied by various structures and contents that pose challenges for machine comprehension. Previous methods generally design task-specific architectures and objectives for individual tasks, resulting in modal isolation and intricate workflows. In this paper, we present a novel large vision-language model, TabPedia, equipped with a concept synergy mechanism. In this mechanism, all the involved diverse visual table understanding (VTU) tasks and multi-source visual embeddings are abstracted as concepts. This unified framework allows TabPedia to seamlessly integrate VTU tasks, such as table detection, table structure recognition, table querying, and table question answering, by leveraging the capabilities of large language models (LLMs). Moreover, the concept synergy mechanism enables table perception-related and comprehension-related tasks to work in harmony, as they can effectively leverage the needed clues from the corresponding source perception embeddings. Furthermore, to better evaluate the VTU task in real-world scenarios, we establish a new and comprehensive table VQA benchmark, ComTQA, featuring approximately 9,000 QA pairs. Extensive quantitative and qualitative experiments on both table perception and comprehension tasks, conducted across various public benchmarks, validate the effectiveness of our TabPedia. The superior performance further confirms the feasibility of using LLMs for understanding visual tables when all concepts work in synergy. The benchmark ComTQA has been open-sourced at https://huggingface.co/datasets/ByteDance/ComTQA. The source code and model also have been released athttps://github.com/zhaowc-ustc/TabPedia.