From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Models

TL;DR

This survey reviews recent advancements in automatic chart understanding driven by large foundation models, highlighting challenges, modeling strategies, and future research directions in the field.

Contribution

It provides a comprehensive overview of recent developments, challenges, and future directions in chart understanding using large foundation models, including evaluation metrics and modeling approaches.

Findings

State-of-the-art performance in chart understanding tasks

Identification of key challenges like domain-specific charts and evaluation metrics

Discussion of tool augmentation techniques for improved performance

Abstract

Data visualization in the form of charts plays a pivotal role in data analysis, offering critical insights and aiding in informed decision-making. Automatic chart understanding has witnessed significant advancements with the rise of large foundation models in recent years. Foundation models, such as large language models, have revolutionized various natural language processing tasks and are increasingly being applied to chart understanding tasks. This survey paper provides a comprehensive overview of the recent developments, challenges, and future directions in chart understanding within the context of these foundation models. We review fundamental building blocks crucial for studying chart understanding tasks. Additionally, we explore various tasks and their evaluation metrics and sources of both charts and textual inputs. Various modeling strategies are then examined, encompassing both classification-based and generation-based approaches, along with tool augmentation techniques that enhance chart understanding performance. Furthermore, we discuss the state-of-the-art performance of each task and discuss how we can improve the performance. Challenges and future directions are addressed, highlighting the importance of several topics, such as domain-specific charts, lack of efforts in developing evaluation metrics, and agent-oriented settings. This survey paper serves as a comprehensive resource for researchers and practitioners in the fields of natural language processing, computer vision, and data analysis, providing valuable insights and directions for future research in chart understanding leveraging large foundation models. The studies mentioned in this paper, along with emerging new research, will be continually updated at: https://github.com/khuangaf/Awesome-Chart-Understanding.