Do LLMs Work on Charts? Designing Few-Shot Prompts for Chart Question Answering and Summarization
Xuan Long Do, Mohammad Hassanpour, Ahmed Masry, Parsa Kavehzadeh,, Enamul Hoque, Shafiq Joty

TL;DR
This paper introduces PromptChart, a multimodal few-shot prompting framework that enables large language models to effectively perform chart question answering and summarization by incorporating visual information into prompts, achieving state-of-the-art results.
Contribution
It presents a novel multimodal prompting strategy for LLMs to handle chart tasks without fine-tuning, including guidelines for prompt design and visual information injection.
Findings
LLMs can excel on chart tasks with proper prompting
PromptChart achieves state-of-the-art performance on benchmarks
Visual information injection improves LLM understanding of charts
Abstract
A number of tasks have been proposed recently to facilitate easy access to charts such as chart QA and summarization. The dominant paradigm to solve these tasks has been to fine-tune a pretrained model on the task data. However, this approach is not only expensive but also not generalizable to unseen tasks. On the other hand, large language models (LLMs) have shown impressive generalization capabilities to unseen tasks with zero- or few-shot prompting. However, their application to chart-related tasks is not trivial as these tasks typically involve considering not only the underlying data but also the visual features in the chart image. We propose PromptChart, a multimodal few-shot prompting framework with LLMs for chart-related applications. By analyzing the tasks carefully, we have come up with a set of prompting guidelines for each task to elicit the best few-shot performance from…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Natural Language Processing Techniques · Advanced Text Analysis Techniques
MethodsSparse Evolutionary Training
