Learning from Teaching Assistants to Program with Subgoals: Exploring the Potential for AI Teaching Assistants

TL;DR

This study explores the use of generative AI as teaching assistants in programming education, showing that AI TAs can help learners solve tasks efficiently and are perceived similarly to human TAs.

Contribution

It demonstrates the feasibility of AI TAs in programming education and provides guidelines for their effective design and use based on empirical user interaction data.

Findings

Learners solved programming tasks faster with AI TAs.

Perception of AI TAs matched that of human TAs in key aspects.

Guidelines for designing effective AI TAs were proposed.

Abstract

With recent advances in generative AI, conversational models like ChatGPT have become feasible candidates for TAs. We investigate the practicality of using generative AI as TAs in introductory programming education by examining novice learners' interaction with TAs in a subgoal learning environment. To compare the learners' interaction and perception of the AI and human TAs, we conducted a between-subject study with 20 novice programming learners. Learners solve programming tasks by producing subgoals and subsolutions with the guidance of a TA. Our study shows that learners can solve tasks faster with comparable scores with AI TAs. Learners' perception of the AI TA is on par with that of human TAs in terms of speed and comprehensiveness of the replies and helpfulness, difficulty, and satisfaction of the conversation. Finally, we suggest guidelines to better design and utilize generative AI as TAs in programming education from the result of our chat log analysis.