Beyond Description: A Multimodal Agent Framework for Insightful Chart Summarization

TL;DR

This paper introduces a novel multi-agent framework that leverages multimodal large language models to generate insightful, deep summaries of charts, surpassing traditional low-level description methods.

Contribution

The paper presents Chart Insight Agent Flow, a new multi-agent approach for chart summarization, and introduces ChartSummInsights, a benchmark dataset with expert-annotated insightful summaries.

Findings

Significant improvement in summarization quality with deep insights.

Effective utilization of MLLMs for complex data interpretation.

Introduction of a new dataset for benchmarking chart summarization.

Abstract

Chart summarization is crucial for enhancing data accessibility and the efficient consumption of information. However, existing methods, including those with Multimodal Large Language Models (MLLMs), primarily focus on low-level data descriptions and often fail to capture the deeper insights which are the fundamental purpose of data visualization. To address this challenge, we propose Chart Insight Agent Flow, a plan-and-execute multi-agent framework effectively leveraging the perceptual and reasoning capabilities of MLLMs to uncover profound insights directly from chart images. Furthermore, to overcome the lack of suitable benchmarks, we introduce ChartSummInsights, a new dataset featuring a diverse collection of real-world charts paired with high-quality, insightful summaries authored by human data analysis experts. Experimental results demonstrate that our method significantly improves the performance of MLLMs on the chart summarization task, producing summaries with deep and diverse insights.