Evaluating Learner Representations for Differentiation Prior to Instructional Outcomes

TL;DR

This paper introduces a new measure called 'distinctiveness' to evaluate learner representations in educational AI, focusing on their ability to differentiate students without relying on instructional outcomes.

Contribution

It proposes a task-agnostic, representation-level evaluation method for learner differentiation, validated through analysis of student interaction data in an online learning environment.

Findings

Learner-level representations show higher separation than interaction-level representations.

Representations based on individual questions outperform aggregated interaction representations.

Distinctiveness effectively assesses the potential for personalized educational modeling.

Abstract

Learner representations play a central role in educational AI systems, yet it is often unclear whether they preserve meaningful differences between students when instructional outcomes are unavailable or highly context-dependent. This work examines how to evaluate learner representations based on whether they retain separation between learners under a shared comparison rule. We introduce distinctiveness, a representation-level measure that evaluates how each learner differs from others in the cohort using pairwise distances, without requiring clustering, labels, or task-specific evaluation. Using student-authored questions collected through a conversational AI agent in an online learning environment, we compare representations based on individual questions with representations that aggregate patterns across a student's interactions over time. Results show that learner-level representations yield higher separation, stronger clustering structure, and more reliable pairwise discrimination than interaction-level representations. These findings demonstrate that learner representations can be evaluated independently of instructional outcomes and provide a practical pre-deployment criterion using distinctiveness as a diagnostic metric for assessing whether a representation supports differentiated modeling or personalization.