Designing a Meta-Reflective Dashboard for Instructor Insight into Student-AI Interactions

TL;DR

This paper introduces a meta-reflective dashboard that provides instructors with interpretable summaries of student-AI interactions, enhancing visibility and understanding while respecting privacy and reducing surveillance concerns.

Contribution

The paper presents a novel dashboard design that summarizes student-AI sessions without exposing raw chat logs, addressing privacy and scalability issues in instructor oversight.

Findings

Reduces instructors' effort in understanding student-AI interactions

Mitigates privacy concerns compared to transcript access

Provides insights for scalable class-level analytics

Abstract

Generative AI tools are increasingly used for coursework help, shifting much of students' help-seeking and reasoning into student-AI chats that are largely invisible to instructors. This loss of visibility can weaken instructors' ability to understand students' difficulties, ensure alignment with course goals, and uphold course policies. Yet transcript-level access is neither scalable nor ethically straightforward: reading raw chat logs across a class is impractical, and exposing detailed dialogue can raise privacy concerns and chilling effects on help seeking. As a result, instructors face a tension between needing actionable insight and avoiding default surveillance of student conversations. To address this gap, we propose a meta-reflective dashboard that makes student-AI sessions interpretable without exposing raw chat logs by default. After each help-seeking session, a reflection AI produces a structured, session-level summary of the student's interaction trajectory, AI usage patterns, and potential risks. We co-designed the dashboard with instructors and students to surface key challenges and design goals, and conducted a formative evaluation of perceived usefulness, trust in the summaries, and privacy acceptability. Findings suggest that the proposed dashboard can reduce instructors' sensemaking effort while mitigating privacy concerns associated with transcript-level access, and they also yield design implications for evidence, governance, and scalable class-level analytics for AI-supported learning.