An Exploration of Higher Education Course Evaluation by Large Language Models

TL;DR

This paper explores how large language models can automate and improve course evaluations in higher education by analyzing classroom discussions and providing structured, reliable feedback that aligns with expert judgments.

Contribution

It demonstrates the effectiveness of LLMs, especially fine-tuned Llama, in reliably conducting detailed course evaluations at scale, surpassing traditional methods.

Findings

LLMs can reliably evaluate courses at micro and macro levels

Fine-tuning improves evaluation accuracy and consistency

LLM feedback offers actionable insights for teaching improvement

Abstract

Course evaluation plays a critical role in ensuring instructional quality and guiding curriculum development in higher education. However, traditional evaluation methods, such as student surveys, classroom observations, and expert reviews, are often constrained by subjectivity, high labor costs, and limited scalability. With recent advancements in large language models (LLMs), new opportunities have emerged for generating consistent, fine-grained, and scalable course evaluations. This study investigates the use of three representative LLMs for automated course evaluation at both the micro level (classroom discussion analysis) and the macro level (holistic course review). Using classroom interaction transcripts and a dataset of 100 courses from a major institution in China, we demonstrate that LLMs can extract key pedagogical features and generate structured evaluation results aligned with expert judgement. A fine-tuned version of Llama shows superior reliability, producing score distributions with greater differentiation and stronger correlation with human evaluators than its counterparts. The results highlight three major findings: (1) LLMs can reliably perform systematic and interpretable course evaluations at both the micro and macro levels; (2) fine-tuning and prompt engineering significantly enhance evaluation accuracy and consistency; and (3) LLM-generated feedback provides actionable insights for teaching improvement. These findings illustrate the promise of LLM-based evaluation as a practical tool for supporting quality assurance and educational decision-making in large-scale higher education settings.