61A Bot Report: AI Assistants in CS1 Save Students Homework Time and Reduce Demands on Staff. (Now What?)

TL;DR

This paper reports on the development and deployment of a GPT-4-based AI homework assistant in a CS1 course, which reduces student homework time and staff questions, while supporting pedagogical goals.

Contribution

The paper introduces a novel GPT-4-based interactive homework assistant that effectively reduces student homework time and questions in a large CS1 course.

Findings

Significant reduction in homework question rates in course forum.

Students using the Bot spent up to 50% less time on assignments.

Over 2000 students made 100,000 requests across two semesters.

Abstract

LLM-based chatbots enable students to get immediate, interactive help on homework assignments, but even a thoughtfully-designed bot may not serve all pedagogical goals. We report here on the development and deployment of a GPT-4-based interactive homework assistant ("61A Bot") for students in a large CS1 course; over 2000 students made over 100,000 requests of our Bot across two semesters. Our assistant offers one-shot, contextual feedback within the command-line "autograder" students use to test their code. Our Bot wraps student code in a custom prompt that supports our pedagogical goals and avoids providing solutions directly. Analyzing student feedback, questions, and autograder data, we find reductions in homework-related question rates in our course forum, as well as reductions in homework completion time when our Bot is available. For students in the 50th-80th percentile, reductions can exceed 30 minutes per assignment, up to 50% less time than students at the same percentile rank in prior semesters. Finally, we discuss these observations, potential impacts on student learning, and other potential costs and benefits of AI assistance in CS1.