Oversight in Action: Experiences with Instructor-Moderated LLM Responses in an Online Discussion Forum

TL;DR

This paper describes the design, deployment, and evaluation of an instructor-controlled bot in an online forum that drafts responses to student questions, aiming to reduce instructor workload while maintaining oversight of LLM-generated content.

Contribution

It introduces a novel instructor-moderated bot system integrated into online discussion forums to assist with managing LLM responses in computing education.

Findings

Alleviated instructor workload effectively.

Facilitated anonymous student participation.

Identified challenges with complex queries.

Abstract

The integration of large language models (LLMs) into computing education offers many potential benefits to student learning, and several novel pedagogical approaches have been reported in the literature. However LLMs also present challenges, one of the most commonly cited being that of student over-reliance. This challenge is compounded by the fact that LLMs are always available to provide instant help and solutions to students, which can undermine their ability to independently solve problems and diagnose and resolve errors. Providing instructor oversight of LLM-generated content can mitigate this problem, however it is often not practical in real-time learning contexts. Online class discussion forums, which are widely used in computing education, present an opportunity for exploring instructor oversight because they operate asynchronously. Unlike real-time interactions, the discussion forum format aligns with the expectation that responses may take time, making oversight not only feasible but also pedagogically appropriate. In this practitioner paper, we present the design, deployment, and evaluation of a `bot' module that is controlled by the instructor, and integrated into an online discussion forum. The bot assists the instructor by generating draft responses to student questions, which are reviewed, modified, and approved before release. Key features include the ability to leverage course materials, access archived discussions, and publish responses anonymously to encourage open participation. We report our experiences using this tool in a 12-week second-year software engineering course on object-oriented programming. Instructor feedback confirmed the tool successfully alleviated workload but highlighted a need for improvement in handling complex, context-dependent queries. We report the features that were viewed as most beneficial, and suggest avenues for future exploration.