ChartCards: A Chart-Metadata Generation Framework for Multi-Task Chart Understanding

TL;DR

This paper introduces ChartCards, a framework for generating comprehensive chart metadata to facilitate multi-task chart understanding, reducing data collection costs and improving model performance across various chart-related tasks.

Contribution

We propose a unified metadata generation framework, ChartCards, enabling multi-task chart understanding and the creation of MetaChart, a large high-quality dataset for training and evaluation.

Findings

Fine-tuning models on MetaChart improves performance by 5% on average.

Notable gains in text-to-chart retrieval and chart-to-table tasks.

Long-CLIP and Llama 3.2-11B achieve 17% and 28% improvements respectively.

Abstract

The emergence of Multi-modal Large Language Models (MLLMs) presents new opportunities for chart understanding. However, due to the fine-grained nature of these tasks, applying MLLMs typically requires large, high-quality datasets for task-specific fine-tuning, leading to high data collection and training costs. To address this, we propose ChartCards, a unified chart-metadata generation framework for multi-task chart understanding. ChartCards systematically synthesizes various chart information, including data tables, visualization code, visual elements, and multi-dimensional semantic captions. By structuring this information into organized metadata, ChartCards enables a single chart to support multiple downstream tasks, such as text-to-chart retrieval, chart summarization, chart-to-table conversion, chart description, and chart question answering. Using ChartCards, we further construct MetaChart, a large-scale high-quality dataset containing 10,862 data tables, 85K charts, and 170 K high-quality chart captions. We validate the dataset through qualitative crowdsourcing evaluations and quantitative fine-tuning experiments across various chart understanding tasks. Fine-tuning six different models on MetaChart resulted in an average performance improvement of 5% across all tasks. The most notable improvements are seen in text-to-chart retrieval and chart-to-table tasks, with Long-CLIP and Llama 3.2-11B achieving improvements of 17% and 28%, respectively.