Tell Me Why: Designing an Explainable LLM-based Dialogue System for Student Problem Behavior Diagnosis

TL;DR

This paper introduces an explainable dialogue system based on fine-tuned LLMs that provides transparent reasoning for student behavior diagnosis, enhancing trust among teachers.

Contribution

It presents a hierarchical attribution method using explainable AI to generate natural-language explanations for LLM-based educational dialogue systems.

Findings

The method outperformed baselines in identifying supporting evidence.

Teachers reported higher trust when explanations were provided.

Preliminary user study with 22 teachers showed increased trust with explanations.

Abstract

Diagnosing student problem behaviors requires teachers to synthesize multifaceted information, identify behavioral categories, and plan intervention strategies. Although fine-tuned large language models (LLMs) can support this process through multi-turn dialogue, they rarely explain why a strategy is recommended, limiting transparency and teachers' trust. To address this issue, we present an explainable dialogue system built on a fine-tuned LLM. The system uses a hierarchical attribution method based on explainable AI (xAI) to identify dialogue evidence for each recommendation and generate a natural-language explanation based on that evidence. In technical evaluation, the method outperformed baseline approaches in identifying supporting evidence. In a preliminary user study with 22 pre-service teachers, participants who received explanations reported higher trust in the system. These findings suggest a promising direction for improving LLM explainability in educational dialogue systems.