Explaining Math Word Problem Solvers

TL;DR

This paper investigates whether neural network-based math word problem solvers truly understand problem semantics or rely on superficial patterns, revealing they often ignore meaningful input cues.

Contribution

It introduces a method to test model reliance on input parts by removing words and shows models often succeed without understanding the problem's semantic content.

Findings

Models are insensitive to word removal, maintaining accuracy with nonsense questions.

Solvers rely on superficial cues rather than semantic understanding.

Potential overfitting to specific words rather than problem logic.

Abstract

Automated math word problem solvers based on neural networks have successfully managed to obtain 70-80\% accuracy in solving arithmetic word problems. However, it has been shown that these solvers may rely on superficial patterns to obtain their equations. In order to determine what information math word problem solvers use to generate solutions, we remove parts of the input and measure the model's performance on the perturbed dataset. Our results show that the model is not sensitive to the removal of many words from the input and can still manage to find a correct answer when given a nonsense question. This indicates that automatic solvers do not follow the semantic logic of math word problems, and may be overfitting to the presence of specific words.