Can LLM Reasoning Be Trusted? A Comparative Study: Using Human Benchmarking on Statistical Tasks

TL;DR

This study evaluates the reasoning and self-assessment abilities of fine-tuned large language models on statistical tasks, comparing their performance to human benchmarks and highlighting their potential in education and research validation.

Contribution

It introduces fine-tuning of open-source LLMs for statistical reasoning and demonstrates their improved performance and self-evaluation capabilities compared to traditional metrics.

Findings

Fine-tuned models perform at a level comparable to statistics students.

Architecture-dependent improvements are observed with some models showing significant gains.

LLMs can effectively assess answer quality, surpassing traditional evaluation metrics.

Abstract

This paper investigates the ability of large language models (LLMs) to solve statistical tasks, as well as their capacity to assess the quality of reasoning. While state-of-the-art LLMs have demonstrated remarkable performance in a range of NLP tasks, their competence in addressing even moderately complex statistical challenges is not well understood. We have fine-tuned selected open-source LLMs on a specially developed dataset to enhance their statistical reasoning capabilities, and compared their performance with the human scores used as a benchmark. Our results show that the fine-tuned models achieve better performance on advanced statistical tasks on the level comparable to a statistics student. Fine-tuning demonstrates architecture-dependent improvements, with some models showing significant performance gains, indicating clear potential for deployment in educational technology and statistical analysis assistance systems. We also show that LLMs themselves can be far better judges of the answers quality (including explanation and reasoning assessment) in comparison to traditional metrics, such as BLEU or BertScore. This self-evaluation capability enables scalable automated assessment for statistical education platforms and quality assurance in automated analysis tools. Potential applications also include validation tools for research methodology in academic and industry settings, and quality control mechanisms for data analysis workflows.