ChartCap: Mitigating Hallucination of Dense Chart Captioning
Junyoung Lim, Jaewoo Ahn, Gunhee Kim

TL;DR
This paper introduces ChartCap, a large-scale dataset of 565K real-world chart images with dense, accurate captions designed to reduce hallucinations in chart captioning, and proposes a new metric for evaluating caption quality.
Contribution
The paper presents a novel dataset, ChartCap, with high-quality, dense captions for charts, and a new evaluation metric, Visual Consistency Score, to improve chart captioning models.
Findings
Models trained on ChartCap produce more accurate, less hallucinated captions.
The Visual Consistency Score correlates well with caption quality.
Fine-tuning on ChartCap outperforms existing datasets and models.
Abstract
Generating accurate, informative, and hallucination-free captions for charts remains challenging for vision language models, primarily due to the lack of large-scale, high-quality datasets of real-world charts. However, existing real-world chart datasets suffer from the inclusion of extraneous information that cannot be inferred from the chart and failure to sufficiently capture structural elements and key insights. Therefore, we introduce ChartCap, a large-scale dataset of 565K real-world chart images paired with type-specific, dense captions that exclude extraneous information and highlight both structural elements and key insights in detail. To build ChartCap, we design a four-stage pipeline that generates captions using only the discernible data from the chart and employ a cycle consistency-based human verification, which accelerates quality control without sacrificing accuracy.…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
