Adversarial Examples for Evaluating Math Word Problem Solvers

TL;DR

This paper evaluates the robustness of math word problem solvers by generating adversarial examples through question reordering and paraphrasing, revealing their sensitivity to linguistic variations and exposing limitations in their understanding.

Contribution

The paper introduces two novel adversarial attack methods for assessing math word problem solvers and demonstrates their effectiveness in significantly reducing solver accuracy.

Findings

Adversarial attacks reduce solver accuracy by over 40 percentage points

Existing solvers are sensitive to linguistic variations in problem text

Generated adversarial examples are validated through human evaluation

Abstract

Standard accuracy metrics have shown that Math Word Problem (MWP) solvers have achieved high performance on benchmark datasets. However, the extent to which existing MWP solvers truly understand language and its relation with numbers is still unclear. In this paper, we generate adversarial attacks to evaluate the robustness of state-of-the-art MWP solvers. We propose two methods Question Reordering and Sentence Paraphrasing to generate adversarial attacks. We conduct experiments across three neural MWP solvers over two benchmark datasets. On average, our attack method is able to reduce the accuracy of MWP solvers by over 40 percentage points on these datasets. Our results demonstrate that existing MWP solvers are sensitive to linguistic variations in the problem text. We verify the validity and quality of generated adversarial examples through human evaluation.