Pre-Calc: Learning to Use the Calculator Improves Numeracy in Language Models

TL;DR

Pre-Calc introduces a pre-finetuning method enabling smaller language models to effectively learn calculator usage, significantly enhancing their numerical comprehension and reasoning capabilities across various datasets.

Contribution

It presents a novel pre-finetuning objective for encoder-only and encoder-decoder models to improve numerical reasoning by learning calculator use.

Findings

Improved performance on numerical reasoning tasks.

Effective pre-finetuning for smaller models.

Enhanced numerical understanding in downstream tasks.

Abstract

Quantitative and numerical comprehension in language is an important task in many fields like education and finance, but still remains a challenging task for language models. While tool and calculator usage has shown to be helpful to improve mathematical reasoning in large pretrained decoder-only language models, this remains unexplored for smaller language models with encoders. In this paper, we propose Pre-Calc, a simple pre-finetuning objective of learning to use the calculator for both encoder-only and encoder-decoder architectures, formulated as a discriminative and generative task respectively. We pre-train BERT and RoBERTa for discriminative calculator use and Flan-T5 for generative calculator use on the MAWPS, SVAMP, and AsDiv-A datasets, which improves performance on downstream tasks that require numerical understanding. Our code and data are available at https://github.com/calc-cmu/pre-calc.