Evaluation of LLM-based Explanations for a Learning Analytics Dashboard

TL;DR

This study evaluates the effectiveness of large language model-generated explanations in learning analytics dashboards, finding they improve user preference and support self-regulated learning in educational settings.

Contribution

It introduces the use of LLMs for generating explanations in learning dashboards and compares their effectiveness to human explanations in an educational context.

Findings

LLM explanations are significantly more favored by users.

LLM explanations support self-regulated learning and reflection.

They maintain pedagogical standards approved by teachers.

Abstract

Learning Analytics Dashboards can be a powerful tool to support self-regulated learning in Digital Learning Environments and promote development of meta-cognitive skills, such as reflection. However, their effectiveness can be affected by the interpretability of the data they provide. To assist in the interpretation, we employ a large language model to generate verbal explanations of the data in the dashboard and evaluate it against a standalone dashboard and explanations provided by human teachers in an expert study with university level educators (N=12). We find that the LLM-based explanations of the skill state presented in the dashboard, as well as general recommendations on how to proceed with learning within the course are significantly more favored compared to the other conditions. This indicates that using LLMs for interpretation purposes can enhance the learning experience for learners while maintaining the pedagogical standards approved by teachers.