Towards AI Agents for Course Instruction in Higher Education: Early Experiences from the Field

TL;DR

This paper explores the deployment of a Large Language Model-driven AI instructor in a graduate course, analyzing student interactions and engagement to assess its effectiveness in supporting scalable, high-quality higher education.

Contribution

It introduces a pedagogical and analytical framework for integrating and evaluating AI agents as instructors in real classroom settings.

Findings

Students actively engage with the AI agent for concept exploration and doubt clarification.

Engagement patterns evolve from broad to focused inquiry across modules.

The framework enables reproducible assessment of AI-student interactions in authentic classrooms.

Abstract

This article presents early findings from designing, deploying and evaluating an AI-based educational agent deployed as the primary instructor in a graduate-level Cloud Computing course at IISc. We detail the design of a Large Language Model (LLM)-driven Instructor Agent, and introduce a pedagogical framework that integrates the Instructor Agent into the course workflow for actively interacting with the students for content delivery, supplemented by the human instructor to offer the course structure and undertake question--answer sessions. We also propose an analytical framework that evaluates the Agent--Student interaction transcripts using interpretable engagement metrics of topic coverage, topic depth and turn-level elaboration. We report early experiences on how students interact with the Agent to explore concepts, clarify doubts and sustain inquiry-driven dialogue during live classroom sessions. We also report preliminary analysis on our evaluation metrics applied across two successive instructional modules that reveals patterns of engagement evolution, transitioning from broad conceptual exploration to deeper, focused inquiry. These demonstrate how structured integration of conversational AI agents can foster reflective learning, offer a reproducible methodology for studying engagement in authentic classroom settings, and support scalable, high-quality higher education.