Integrating AI Tutors in a Programming Course
Iris Ma, Alberto Krone Martins, Cristina Videira Lopes

TL;DR
This paper presents RAGMan, an LLM-powered tutoring system integrated into an introductory programming course, demonstrating high accuracy and positive student feedback in supporting homework and learning.
Contribution
Introduces RAGMan, a novel retrieval-augmented LLM tutoring system tailored for course-specific support, with deployment and evaluation in a real university setting.
Findings
98% accuracy in responding to legitimate homework questions
78% of students reported improved learning experience
Half of the students engaged with the AI tutors
Abstract
RAGMan is an LLM-powered tutoring system that can support a variety of course-specific and homework-specific AI tutors. RAGMan leverages Retrieval Augmented Generation (RAG), as well as strict instructions, to ensure the alignment of the AI tutors' responses. By using RAGMan's AI tutors, students receive assistance with their specific homework assignments without directly obtaining solutions, while also having the ability to ask general programming-related questions. RAGMan was deployed as an optional resource in an introductory programming course with an enrollment of 455 students. It was configured as a set of five homework-specific AI tutors. This paper describes the interactions the students had with the AI tutors, the students' feedback, and a comparative grade analysis. Overall, about half of the students engaged with the AI tutors, and the vast majority of the interactions were…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsIntelligent Tutoring Systems and Adaptive Learning · Online Learning and Analytics
MethodsSparse Evolutionary Training
