RealChart2Code: Advancing Chart-to-Code Generation with Real Data and Multi-Task Evaluation

TL;DR

RealChart2Code introduces a large-scale benchmark for evaluating vision-language models on complex, real-world chart generation tasks, highlighting current limitations and guiding future research.

Contribution

It is the first benchmark to systematically evaluate chart generation from authentic data and iterative code refinement in a conversational setting.

Findings

Significant performance drop of VLMs on complex, real-world charts.

Large performance gap between proprietary and open-weight models.

State-of-the-art VLMs often fail to accurately generate multi-panel charts.

Abstract

Vision-Language Models (VLMs) have demonstrated impressive capabilities in code generation across various domains. However, their ability to replicate complex, multi-panel visualizations from real-world data remains largely unassessed. To address this gap, we introduce \textbf{\texttt{RealChart2Code}}, a new large-scale benchmark with over 2,800 instances grounded in authentic datasets and featuring tasks with clear analytical intent. Crucially, it is the first benchmark to systematically evaluate chart generation from large-scale raw data and assess iterative code refinement in a multi-turn conversational setting. Our comprehensive evaluation of 14 leading VLMs on \texttt{RealChart2Code} reveals significant performance degradation compared to simpler benchmarks, highlighting their struggles with complex plot structures and authentic data. Our analysis uncovers a substantial performance gap between proprietary and open-weight models and confirms that even state-of-the-art VLMs often fail to accurately replicate intricate, multi-panel charts. These findings provide valuable insights into the current limitations of VLMs and guide future research directions. We release the benchmark and code at \url{https://github.com/Speakn0w/RealChart2Code}.