CS-Guide: Leveraging LLMs and Student Reflections to Provide Frequent, Scalable Academic Monitoring Feedback to Computer Science Students

TL;DR

CS-Guide utilizes large language models and student reflections to provide scalable, timely academic feedback and interventions for computer science students, addressing the challenge of limited advising resources.

Contribution

The paper introduces CS-Guide, a novel system that combines LLMs and learning analytics to deliver personalized, frequent academic support using student-generated data.

Findings

Achieved up to 97% F1 score in intervention recommendations.

Demonstrated effective integration of LLMs with learning analytics.

Enhanced advising systems with scalable, domain-specific feedback.

Abstract

Computer Science (CS) departments often serve large student populations, making timely academic monitoring and personalized feedback difficult. While the recommended counselor-to-student ratio is 250:1, it often exceeds 350:1 in practice, leading to delays in support and interventions. We present CS-Guide, which leverages Large Language Models (LLMs) to deliver scalable, frequent academic feedback. Weekly, students interact with CS-Guide through self-reported grades and reflective journal entries, from which CS-Guide extracts quantitative and qualitative features and triggers tailored interventions (e.g., academic support, health and wellness referrals). Thus, CS-Guide uniquely integrates learning analytics, LLMs, and actionable interventions using both structured and unstructured student-generated data. We evaluated CS-Guide on a four-year, ~20K-entry longitudinal dataset, and it achieved up to a 97% F1 score in recommending interventions for first-year students. This shows that CS-Guide can enhance advising systems with scalable, consistent, timely, and domain-specific feedback.