AI-Assisted Code Review as a Scaffold for Code Quality and Self-Regulated Learning: An Experience Report

TL;DR

This study explores integrating an AI-assisted code review tool into GitHub for software engineering students, demonstrating increased engagement, stable responsiveness, and educational benefits in authentic project settings.

Contribution

It introduces a workflow design for AI reviewers that supports learning, compares two student cohorts, and offers pedagogical insights for responsible AI use in education.

Findings

2024 cohort showed more iterative PR activity (1176 vs. 581)

Technical issues decreased to zero after refinements

Responsiveness to AI reviews remained stable (~33%) across cohorts

Abstract

Code review is central to software engineering education but hard to scale in capstone projects due to tight deadlines, uneven peer feedback, and limited prior experience. We investigate an LLM-as-reviewer integrated directly into GitHub pull requests (human-in-the-loop) across two cohorts (more than 100 students, 2023--2024). Using a mixed-methods design -- GitHub data, reflective reports, and a targeted survey -- we examine engagement and responsiveness as behavioral indicators of self-regulated learning processes. Quantitatively, the 2024 cohort produced more iterative activity (1176 vs. 581 PRs), while technical issues observed in 2023 (227 failed AI attempts) dropped to zero after tool and instructional refinements. Despite different adoption levels (93\% vs. 50\% of teams using the tool), responsiveness was stable: 32\% (2023) and 33\% (2024) of successfully AI-reviewed PRs were followed by subsequent commits on the same PR. Qualitatively, students used the LLM's structured comments to focus reviews and discuss code quality, while guidance reduced over-reliance. We contribute: (i) an in-workflow design for an AI reviewer that scaffolds learning while mitigating cognitive offloading; (ii) a repeated cross sectional comparison across two cohorts in authentic settings; (iii) a mixed-methods analysis combining objective GitHub metrics with student self-reports; and (iv) evidence-based pedagogical recommendations for responsible, student-led AI-assisted review.