Cutting Through the Noise: Boosting LLM Performance on Math Word Problems

TL;DR

This paper introduces a prompting framework and adversarial training dataset to improve large language models' robustness in solving math word problems with irrelevant information, demonstrating enhanced performance and generalizability.

Contribution

It presents a novel adversarial dataset and a prompting framework to boost LLM robustness against irrelevant information in math word problems.

Findings

LLMs' performance drops by ~26% on adversarial MWPs without mitigation.

Fine-tuning on adversarial samples improves LLM performance by ~8%.

LLMs' performance on adversarial GSM-8K-Adv decreases by up to 6%.

Abstract

Large Language Models (LLMs) excel at various tasks, including solving math word problems (MWPs), but struggle with real-world problems containing irrelevant information. To address this, we propose a prompting framework that generates adversarial variants of MWPs by adding irrelevant variables. We introduce a dataset, PROBLEMATHIC, containing both adversarial and non-adversarial MWPs. Our experiments reveal that LLMs are susceptible to distraction by numerical noise, resulting in an average relative performance drop of ~26% on adversarial MWPs. To mitigate this, we fine-tune LLMs (Llama-2, Mistral) on the adversarial samples from our dataset. Fine-tuning on adversarial training instances improves performance on adversarial MWPs by ~8%, indicating increased robustness to noise and improved ability to identify relevant data for reasoning. Finally, to assess the generalizability of our prompting framework, we introduce GSM-8K-Adv, an adversarial variant of the GSM-8K benchmark. LLMs continue to struggle when faced with adversarial information, reducing performance by up to 6%.