Integrating Large Language Models and Evaluating Student Outcomes in an Introductory Computer Science Course

TL;DR

This study explores how integrating Large Language Models into an introductory CS course affects student outcomes, showing similar exam performance to traditional courses and highlighting the value of open-ended projects and student perceptions.

Contribution

Introduces a new CS1 course design that incorporates LLMs as learning tools and evaluates its impact on student performance and perceptions.

Findings

Student exam outcomes are comparable to traditional courses.

Open-ended projects are especially beneficial with LLM integration.

Most students find LLM tools helpful, with some concerns about over-reliance.

Abstract

Generative AI (GenAI) models have broad implications for education in general, impacting the foundations of what we teach and how we assess. This is especially true in computing, where LLMs tuned for coding have demonstrated shockingly good performance on the types of assignments historically used in introductory CS (CS1) courses. As a result, CS1 courses will need to change what skills are taught and how they are assessed. Computing education researchers have begun to study student use of LLMs, but there remains much to be understood about the ways that these tools affect student outcomes. In this paper, we present the design and evaluation of a new CS1 course at a large research-intensive university that integrates the use of LLMs as a learning tool for students. We describe the design principles used to create our new CS1-LLM course, our new course objectives, and evaluation of student outcomes and perceptions throughout the course as measured by assessment scores and surveys. Our findings suggest that 1) student exam performance outcomes, including differences among demographic groups, are largely similar to historical outcomes for courses without integration of LLM tools, 2) large, open-ended projects may be particularly valuable in an LLM context, and 3) students predominantly found the LLM tools helpful, although some had concerns regarding over-reliance on the tools.