Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities

TL;DR

This paper investigates the challenges data scientists face when interacting with LLM-powered AI tools in data science, identifying obstacles and proposing design improvements to enhance usability and effectiveness.

Contribution

It provides a comprehensive analysis of conversational obstacles in AI-powered data science and offers actionable design recommendations to address these issues.

Findings

Key challenges include contextual data retrieval and prompt formulation.

Data brushing and feedback loops are effective design solutions.

Insights are based on mixed-methods study with practitioners.

Abstract

Large Language Models (LLMs) are being increasingly employed in data science for tasks like data preprocessing and analytics. However, data scientists encounter substantial obstacles when conversing with LLM-powered chatbots and acting on their suggestions and answers. We conducted a mixed-methods study, including contextual observations, semi-structured interviews (n=14), and a survey (n=114), to identify these challenges. Our findings highlight key issues faced by data scientists, including contextual data retrieval, formulating prompts for complex tasks, adapting generated code to local environments, and refining prompts iteratively. Based on these insights, we propose actionable design recommendations, such as data brushing to support context selection, and inquisitive feedback loops to improve communications with AI-based assistants in data-science tools.