Evaluating Prompting Strategies for Chart Question Answering with Large Language Models

TL;DR

This study systematically evaluates prompting strategies for chart question answering using large language models, revealing that Few-Shot Chain-of-Thought prompts significantly enhance reasoning accuracy on structured chart data.

Contribution

It provides a comprehensive comparison of prompting paradigms across multiple models and introduces a framework isolating prompt structure as the key variable.

Findings

Few-Shot Chain-of-Thought prompts achieve up to 78.2% accuracy.

Few-Shot prompting improves format adherence.

Zero-Shot performs well only on simpler tasks with high-capacity models.

Abstract

Prompting strategies affect LLM reasoning performance, but their role in chart-based QA remains underexplored. We present a systematic evaluation of four widely used prompting paradigms (Zero-Shot, Few-Shot, Zero-Shot Chain-of-Thought, and Few-Shot Chain-of-Thought) across GPT-3.5, GPT-4, and GPT-4o on the ChartQA dataset. Our framework operates exclusively on structured chart data, isolating prompt structure as the only experimental variable, and evaluates performance using two metrics: Accuracy and Exact Match. Results from 1,200 diverse ChartQA samples show that Few-Shot Chain-of-Thought prompting consistently yields the highest accuracy (up to 78.2\%), particularly on reasoning-intensive questions, while Few-Shot prompting improves format adherence. Zero-Shot performs well only with high-capacity models on simpler tasks. These findings provide actionable guidance for selecting prompting strategies in structured data reasoning tasks, with implications for both efficiency and accuracy in real-world applications.