Designing Prompt Analytics Dashboards to Analyze Student-ChatGPT Interactions in EFL Writing

TL;DR

This paper presents the design and development of a Prompt Analytics Dashboard (PAD) to help teachers analyze student-ChatGPT interactions in EFL writing classes, addressing risks like inaccuracies, biases, and over-reliance.

Contribution

It introduces a novel dashboard prototype that integrates chat logs and editing history, informed by teacher feedback, to enhance analysis and teaching strategies.

Findings

Teacher feedback improved dashboard features

PAD enables analysis of student behaviors and learning loops

Supports teachers in developing teaching skills

Abstract

While ChatGPT has significantly impacted education by offering personalized resources for students, its integration into educational settings poses unprecedented risks, such as inaccuracies and biases in AI-generated content, plagiarism and over-reliance on AI, and privacy and security issues. To help teachers address such risks, we conducted a two-phase iterative design process that comprises surveys, interviews, and prototype demonstration involving six EFL (English as a Foreign Language) teachers, who integrated ChatGPT into semester-long English essay writing classes. Based on the needs identified during the initial survey and interviews, we developed a prototype of Prompt Analytics Dashboard (PAD) that integrates the essay editing history and chat logs between students and ChatGPT. Teacher's feedback on the prototype informs additional features and unmet needs for designing future PAD, which helps them (1) analyze contextual analysis of student behaviors, (2) design an overall learning loop, and (3) develop their teaching skills.