Why Do Open-Source LLMs Struggle with Data Analysis? A Systematic Empirical Study

TL;DR

This paper systematically evaluates open-source LLMs' data analysis abilities, identifying key factors affecting performance and proposing a data synthesis method to improve reasoning in complex, reasoning-intensive tasks.

Contribution

It introduces a comprehensive evaluation framework and a novel data synthesis approach to enhance open-source LLMs' data analysis capabilities.

Findings

Strategic planning quality is the main performance factor.

Interaction design and task complexity affect reasoning.

Data quality impacts performance more than data diversity.

Abstract

Large Language Models (LLMs) hold promise in automating data analysis tasks, yet open-source models face significant limitations in these kinds of reasoning-intensive scenarios. In this work, we investigate strategies to enhance the data analysis capabilities of open-source LLMs. By curating a seed dataset of diverse, realistic scenarios, we evaluate model behavior across three core dimensions: data understanding, code generation, and strategic planning. Our analysis reveals three key findings: (1) Strategic planning quality serves as the primary determinant of model performance; (2) Interaction design and task complexity significantly influence reasoning capabilities; (3) Data quality demonstrates a greater impact than diversity in achieving optimal performance. We leverage these insights to develop a data synthesis methodology, demonstrating significant improvements in open-source LLMs' analytical reasoning capabilities. Code is available at https://github.com/zjunlp/DataMind.