Learning Context Matters: Measuring and Diagnosing Personalization Gaps in LLM-Based Instructional Design

TL;DR

This paper presents a framework for measuring and diagnosing how learning context influences LLM-based instructional decisions, revealing that while context impacts decisions, significant misalignments with expert pedagogical judgments remain.

Contribution

It introduces a novel framework for evaluating and diagnosing personalization gaps in LLM-based tutoring systems based on learning context and expert comparison.

Findings

Context-aware LLM decisions are closer to expert judgments.

Significant misalignment persists despite context inclusion.

Relevance-impact analysis identifies influential learner characteristics.

Abstract

The adoption of generative AI in education has accelerated dramatically in recent years, with Large Language Models (LLMs) increasingly integrated into learning environments in the hope of providing personalized support that enhances learner engagement and knowledge retention. However, truly personalized support requires access to meaningful Learning Context (LC) regarding who the learner is, what they are trying to understand, and how they are engaging with the material. In this paper, we present a framework for measuring and diagnosing how the LC influences instructional strategy selection in LLM-based tutoring systems. Using psychometrically grounded synthetic learning contexts and a pedagogically grounded decision space, we compare LLM instructional decisions in context-blind and context-aware conditions and quantify their alignment with the pedagogical judgments of subject matter experts. Our results show that, while providing the LC induces systematic, measurable changes in instructional decisions that move LLM policies closer to the subject matter expert policy, substantial misalignment remains. To diagnose this misalignment, we introduce a relevance-impact analysis that reveals which learner characteristics are attended to, ignored, or spuriously influential in LLM instructional decision-making. This analysis, conducted in collaboration with subject matter experts, demonstrates that LC materially shapes LLM instructional planning but does not reliably induce pedagogically appropriate personalization. Our results enable principled evaluation of context-aware LLM systems and provide a foundation for improving personalization through learner characteristic prioritization, pedagogical model tuning, and LC engineering.