ChartCitor: Multi-Agent Framework for Fine-Grained Chart Visual Attribution

TL;DR

ChartCitor is a multi-agent framework that improves chart question-answering by providing fine-grained visual evidence citations, enhancing explainability and user trust in AI-generated responses.

Contribution

It introduces a novel multi-agent system that accurately grounds LLM responses in chart images through fine-grained bounding box citations, addressing limitations of existing methods.

Findings

Outperforms existing baselines across various chart types.

Enhances user trust and explainability in AI chart QA.

Facilitates professional productivity with better evidence grounding.

Abstract

Large Language Models (LLMs) can perform chart question-answering tasks but often generate unverified hallucinated responses. Existing answer attribution methods struggle to ground responses in source charts due to limited visual-semantic context, complex visual-text alignment requirements, and difficulties in bounding box prediction across complex layouts. We present ChartCitor, a multi-agent framework that provides fine-grained bounding box citations by identifying supporting evidence within chart images. The system orchestrates LLM agents to perform chart-to-table extraction, answer reformulation, table augmentation, evidence retrieval through pre-filtering and re-ranking, and table-to-chart mapping. ChartCitor outperforms existing baselines across different chart types. Qualitative user studies show that ChartCitor helps increase user trust in Generative AI by providing enhanced explainability for LLM-assisted chart QA and enables professionals to be more productive.