MalAlgoQA: Pedagogical Evaluation of Counterfactual Reasoning in Large Language Models and Implications for AI in Education

TL;DR

MalAlgoQA introduces a dataset to evaluate large language models' ability to understand flawed reasoning in educational contexts, revealing challenges in counterfactual reasoning and the impact of prompting techniques.

Contribution

The paper presents MalAlgoQA, a new dataset and evaluation framework for assessing counterfactual reasoning in LLMs, highlighting limitations and effects of prompting methods.

Findings

State-of-the-art LLMs perform worse on malgorithm identification than on correct rationale identification.

Chain-of-thought prompting does not consistently improve counterfactual reasoning performance.

Results have implications for AI tutoring systems and addressing student misconceptions.

Abstract

This paper introduces MalAlgoQA, a novel dataset designed to evaluate the counterfactual reasoning capabilities of Large Language Models (LLMs) through a pedagogical approach. The dataset comprises mathematics and reading comprehension questions, each accompanied by four answer choices and their corresponding rationales. At the heart of MalAlgoQA are ``malgorithms'' - rationales behind incorrect answer choices that represent flawed yet logically coherent reasoning paths. These malgorithms serve as counterfactual scenarios, allowing us to assess an LLM's ability to identify and analyze flawed reasoning patterns. We propose the Malgorithm Identification task, where LLMs are assessed based on their ability to identify corresponding malgorithm given an incorrect answer choice. To evaluate the model performance, we introduce two metrics: Algorithm Identification Accuracy (AIA) for correct answer rationale identification, and Malgorithm Identification Accuracy (MIA) for incorrect answer rationale identification. Our experiments reveal that state-of-the-art LLMs exhibit significant performance drops in MIA compared to AIA, highlighting the challenges in counterfactual reasoning. Surprisingly, we find that the chain-of-thought prompting technique not only fails to consistently enhance MIA but can sometimes lead to underperformance compared to simple prompting. These findings have important implications for developing LLMs with improved counterfactual reasoning, particularly relevant for AI-powered tutoring systems, where identifying and addressing student misconceptions is essential. MalAlgoQA dataset is available \href{https://github.com/luffycodes/MalAlgoQA-Dataset}{here}.