Buggy rule diagnosis for combined steps through final answer evaluation in stepwise tasks

TL;DR

This paper proposes a method for diagnosing errors in stepwise tasks by evaluating the final answer, effectively handling combined steps and reducing diagnostic complexity in intelligent tutoring systems.

Contribution

It introduces a final answer-based error diagnosis approach for combined steps, validated on quadratic equation solving data, showing high accuracy and potential for improved tutoring support.

Findings

Final answer evaluation diagnosed 29.4% of complex steps.

Diagnosis aligned with teachers in 97% of cases.

Approach reduces combinatorial explosion in error diagnosis.

Abstract

Many intelligent tutoring systems can support a student in solving a stepwise task. When a student combines several steps in one step, the number of possible paths connecting consecutive inputs may be very large. This combinatorial explosion makes error diagnosis hard. Using a final answer to diagnose a combination of steps can mitigate the combinatorial explosion, because there are generally fewer possible (erroneous) final answers than (erroneous) solution paths. An intermediate input for a task can be diagnosed by automatically completing it according to the task solution strategy and diagnosing this solution. This study explores the potential of automated error diagnosis based on a final answer. We investigate the design of a service that provides a buggy rule diagnosis when a student combines several steps. To validate the approach, we apply the service to an existing dataset (n=1939) of unique student steps when solving quadratic equations, which could not be diagnosed by a buggy rule service that tries to connect consecutive inputs with a single rule. Results show that final answer evaluation can diagnose 29,4% of these steps. Moreover, a comparison of the generated diagnoses with teacher diagnoses on a subset (n=115) shows that the diagnoses align in 97% of the cases. These results can be considered a basis for further exploration of the approach.