Measuring and Improving BERT's Mathematical Abilities by Predicting the Order of Reasoning

TL;DR

This paper explores BERT's mathematical reasoning abilities, fine-tunes it on math problem datasets, introduces new training tasks to enhance its performance, and achieves significant improvements over baseline models.

Contribution

The paper introduces Reasoning Order Prediction tasks to improve BERT's mathematical reasoning and demonstrates their effectiveness in enhancing model performance.

Findings

BERT can learn mathematical reasoning with proper fine-tuning.

Reasoning Order Prediction tasks improve BERT's accuracy on math problems.

Proposed methods reduce positional bias in language models.

Abstract

Imagine you are in a supermarket. You have two bananas in your basket and want to buy four apples. How many fruits do you have in total? This seemingly straightforward question can be challenging for data-driven language models, even if trained at scale. However, we would expect such generic language models to possess some mathematical abilities in addition to typical linguistic competence. Towards this goal, we investigate if a commonly used language model, BERT, possesses such mathematical abilities and, if so, to what degree. For that, we fine-tune BERT on a popular dataset for word math problems, AQuA-RAT, and conduct several tests to understand learned representations better. Since we teach models trained on natural language to do formal mathematics, we hypothesize that such models would benefit from training on semi-formal steps that explain how math results are derived. To better accommodate such training, we also propose new pretext tasks for learning mathematical rules. We call them (Neighbor) Reasoning Order Prediction (ROP or NROP). With this new model, we achieve significantly better outcomes than data-driven baselines and even on-par with more tailored models. We also show how to reduce positional bias in such models.