From Clicks to Conversations: Evaluating the Effectiveness of Conversational Agents in Statistical Analysis

TL;DR

This study introduces StatZ, a conversational agent for statistical analysis, which outperforms traditional GUI-based software in accuracy, efficiency, and user satisfaction across diverse user backgrounds.

Contribution

The paper presents a novel conversational agent, StatZ, and empirically evaluates its effectiveness against established statistical software, demonstrating significant improvements.

Findings

StatZ significantly reduces task completion time.

StatZ improves accuracy in statistical tasks.

Users report higher satisfaction with StatZ.

Abstract

The rapid proliferation of data science forced different groups of individuals with different backgrounds to adapt to statistical analysis. We hypothesize that conversational agents are better suited for statistical analysis than traditional graphical user interfaces (GUI). In this work, we propose a novel conversational agent, StatZ, for statistical analysis. We evaluate the efficacy of StatZ relative to established statistical software:SPSS, SAS, Stata, and JMP in terms of accuracy, task completion time, user experience, and user satisfaction. We combined the proposed analysis question from state-of-the-art language models with suggestions from statistical analysis experts and tested with 51 participants from diverse backgrounds. Our experimental design assessed each participant's ability to perform statistical analysis tasks using traditional statistical analysis tools with GUI and our conversational agent. Results indicate that the proposed conversational agents significantly outperform GUI statistical software in all assessed metrics, including quantitative (task completion time, accuracy, and user experience), and qualitative (user satisfaction) metrics. Our findings underscore the potential of using conversational agents to enhance statistical analysis processes, reducing cognitive load and learning curves and thereby proliferating data analysis capabilities, to individuals with limited knowledge of statistics.