Hierarchical Apprenticeship Learning from Imperfect Demonstrations with Evolving Rewards

TL;DR

HALIDE is a hierarchical apprenticeship learning method that effectively leverages and ranks imperfect, evolving student demonstrations to better infer pedagogical strategies and reward functions.

Contribution

The paper introduces HALIDE, a novel hierarchical framework that models and ranks imperfect student demonstrations with evolving rewards, improving pedagogical decision prediction.

Findings

HALIDE outperforms existing methods in predicting student pedagogical decisions.

Incorporating demonstration ranking improves reward inference accuracy.

Hierarchical modeling captures higher-level student intent from suboptimal actions.

Abstract

While apprenticeship learning has shown promise for inducing effective pedagogical policies directly from student interactions in e-learning environments, most existing approaches rely on optimal or near-optimal expert demonstrations under a fixed reward. Real-world student interactions, however, are often inherently imperfect and evolving: students explore, make errors, revise strategies, and refine their goals as understanding develops. In this work, we argue that imperfect student demonstrations are not noise to be discarded, but structured signals-provided their relative quality is ranked. We introduce HALIDE, Hierarchical Apprenticeship Learning from Imperfect Demonstrations with Evolving Rewards, which not only leverages sub-optimal student demonstrations, but ranks them within a hierarchical learning framework. HALIDE models student behavior at multiple levels of abstraction, enabling inference of higher-level intent and strategy from suboptimal actions while explicitly capturing the temporal evolution of student reward functions. By integrating demonstration quality into hierarchical reward inference,HALIDE distinguishes transient errors from suboptimal strategies and meaningful progress toward higher-level learning goals. Our results show that HALIDE more accurately predicts student pedagogical decisions than approaches that rely on optimal trajectories, fixed rewards, or unranked imperfect demonstrations.