Do Not Trust a Model Because It is Confident: Uncovering and Characterizing Unknown Unknowns to Student Success Predictors in Online-Based Learning

TL;DR

This paper highlights the importance of detecting unknown unknowns in student success models, which are cases where the model is overconfident but wrong, and introduces a framework to identify and analyze these cases to improve trust and reliability.

Contribution

The paper presents a novel framework for detecting and characterizing unknown unknowns in student success prediction models, addressing a critical trust issue in educational AI applications.

Findings

Unknown unknowns are prevalent and critical in student success models.

The proposed framework effectively detects and characterizes these unknown unknowns.

Analysis of log data and instructor interviews validates the framework's usefulness.

Abstract

Student success models might be prone to develop weak spots, i.e., examples hard to accurately classify due to insufficient representation during model creation. This weakness is one of the main factors undermining users' trust, since model predictions could for instance lead an instructor to not intervene on a student in need. In this paper, we unveil the need of detecting and characterizing unknown unknowns in student success prediction in order to better understand when models may fail. Unknown unknowns include the students for which the model is highly confident in its predictions, but is actually wrong. Therefore, we cannot solely rely on the model's confidence when evaluating the predictions quality. We first introduce a framework for the identification and characterization of unknown unknowns. We then assess its informativeness on log data collected from flipped courses and online courses using quantitative analyses and interviews with instructors. Our results show that unknown unknowns are a critical issue in this domain and that our framework can be applied to support their detection. The source code is available at https://github.com/epfl-ml4ed/unknown-unknowns.