Locating acts of mechanistic reasoning in student team conversations with mechanistic machine learning

TL;DR

This paper presents an interpretable machine learning model that identifies moments of students' mechanistic reasoning in team conversations, aiding STEM education research.

Contribution

It introduces a probabilistic model with an inductive bias that improves generalization in detecting mechanistic reasoning across diverse student transcripts.

Findings

Inductive bias enhances model generalization to new students and contexts.

The model's interpretability is integrated into its design, not added afterward.

Practical recommendations are provided for STEM educators and ML researchers.

Abstract

STEM education researchers are often interested in identifying moments of students' mechanistic reasoning for deeper analysis, but have limited capacity to search through many team conversation transcripts to find segments with a high concentration of such reasoning. We offer a solution in the form of an interpretable machine learning model that outputs time-varying probabilities that individual students are engaging in acts of mechanistic reasoning, leveraging evidence from their own utterances as well as contributions from the rest of the group. Using the toolkit of intentionally-designed probabilistic models, we introduce a specific inductive bias that steers the probabilistic dynamics toward desired, domain-aligned behavior. Experiments compare trained models with and without the inductive bias components, investigating whether their presence improves the desired model behavior on transcripts involving never-before-seen students and a novel discussion context. Our results show that the inductive bias improves generalization -- supporting the claim that interpretability is built into the model for this task rather than imposed post hoc. We conclude with practical recommendations for STEM education researchers seeking to adopt the tool and for ML researchers aiming to extend the model's design. Overall, we hope this work encourages the development of mechanistically interpretable models that are understandable and controllable for both end users and model designers in STEM education research.