MalruleLib: Large-Scale Executable Misconception Reasoning with Step Traces for Modeling Student Thinking in Mathematics

TL;DR

MalruleLib is a scalable framework that models student misconceptions in mathematics by translating them into executable rules, enabling large-scale reasoning, diagnosis, and feedback across diverse problem templates.

Contribution

It introduces MalruleLib, a novel system that formalizes misconception reasoning with executable malrules and step traces, supporting scalable student modeling across numerous problem templates.

Findings

Accuracy drops from 66% to 40% on cross-template misconception prediction.

Generating step traces improves student answer prediction by 3-15%.

MalruleLib can generate over one million instances for scalable evaluation.

Abstract

Student mistakes in mathematics are often systematic: a learner applies a coherent but wrong procedure and repeats it across contexts. We introduce MalruleLib, a learning-science-grounded framework that translates documented misconceptions into executable procedures, drawing on 67 learning-science and mathematics education sources, and generates step-by-step traces of malrule-consistent student work. We formalize a core student-modeling problem as Malrule Reasoning Accuracy (MRA): infer a misconception from one worked mistake and predict the student's next answer under cross-template rephrasing. Across nine language models (4B-120B), accuracy drops from 66% on direct problem solving to 40% on cross-template misconception prediction. MalruleLib encodes 101 malrules over 498 parameterized problem templates and produces paired dual-path traces for both correct reasoning and malrule-consistent student reasoning. Because malrules are executable and templates are parameterizable, MalruleLib can generate over one million instances, enabling scalable supervision and controlled evaluation. Using MalruleLib, we observe cross-template degradations of 10-21%, while providing student step traces improves prediction by 3-15%. We release MalruleLib as infrastructure for educational AI that models student procedures across contexts, enabling diagnosis and feedback that targets the underlying misconception.