Are Large Language Models Good Statisticians?

TL;DR

This paper evaluates the statistical analysis capabilities of Large Language Models using a new benchmark, revealing significant performance gaps and highlighting differences in error types compared to humans.

Contribution

Introduces StatQA, a comprehensive benchmark for assessing LLMs on statistical tasks, and provides systematic analysis of their strengths and weaknesses.

Findings

GPT-4o achieves only 64.83% accuracy on StatQA.

Fine-tuned open-source LLMs outperform in-context learning models.

Humans mainly make statistical task confusion errors, LLMs make applicability errors.

Abstract

Large Language Models (LLMs) have demonstrated impressive capabilities across a range of scientific tasks including mathematics, physics, and chemistry. Despite their successes, the effectiveness of LLMs in handling complex statistical tasks remains systematically under-explored. To bridge this gap, we introduce StatQA, a new benchmark designed for statistical analysis tasks. StatQA comprises 11,623 examples tailored to evaluate LLMs' proficiency in specialized statistical tasks and their applicability assessment capabilities, particularly for hypothesis testing methods. We systematically experiment with representative LLMs using various prompting strategies and show that even state-of-the-art models such as GPT-4o achieve a best performance of only 64.83%, indicating significant room for improvement. Notably, while open-source LLMs (e.g. LLaMA-3) show limited capability, those fine-tuned ones exhibit marked improvements, outperforming all in-context learning-based methods (e.g. GPT-4o). Moreover, our comparative human experiments highlight a striking contrast in error types between LLMs and humans: LLMs primarily make applicability errors, whereas humans mostly make statistical task confusion errors. This divergence highlights distinct areas of proficiency and deficiency, suggesting that combining LLM and human expertise could lead to complementary strengths, inviting further investigation into their collaborative potential. Our source code and data are available at https://statqa.github.io/.