Disentangling Learning from Judgment: Representation Learning for Open Response Analytics

TL;DR

This paper introduces an analytics framework that disentangles student response content from teacher grading tendencies, enabling more transparent and interpretable assessment analytics through modeling teacher effects and content signals.

Contribution

It presents a novel pipeline that separates content from judgment in open responses, improving interpretability and enabling reflection on grading practices using embedding-based models.

Findings

Teacher priors significantly influence grade predictions.

Combining priors with content embeddings yields the best predictive performance.

Adjusting for rater effects enhances feature selection and interpretability.

Abstract

Open-ended responses are central to learning, yet automated scoring often conflates what students wrote with how teachers grade. We present an analytics-first framework that separates content signals from rater tendencies, making judgments visible and auditable via analytics. Using de-identified ASSISTments mathematics responses, we model teacher histories as dynamic priors and represent text with sentence embeddings. We apply centroid normalization and response-problem embedding differences, and explicitly model teacher effects with priors to reduce problem- and teacher-related confounds. Temporally-validated linear models quantify the contributions of each signal, and model disagreements surface observations for qualitative inspection. Results show that teacher priors heavily influence grade predictions; the strongest results arise when priors are combined with content embeddings (AUC~0.815), while content-only models remain above chance but substantially weaker (AUC~0.626). Adjusting for rater effects sharpens the selection of features derived from content representations, retaining more informative embedding dimensions and revealing cases where semantic evidence supports understanding as opposed to surface-level differences in how students respond. The contribution presents a practical pipeline that transforms embeddings from mere features into learning analytics for reflection, enabling teachers and researchers to examine where grading practices align (or conflict) with evidence of student reasoning and learning.