Investigating Numeracy Learning Ability of a Text-to-Text Transfer Model

TL;DR

This paper examines the numeracy learning capabilities of the T5 text-to-text transfer model, revealing its strengths in interpolation but significant challenges in extrapolation across various numeracy tasks.

Contribution

It provides an empirical analysis of T5's numeracy skills, highlighting limitations in extrapolation and suggesting areas for model improvement.

Findings

T5 performs well in interpolation tasks.

T5 struggles with extrapolation in numeracy tasks.

Numeracy understanding remains limited in current models.

Abstract

The transformer-based pre-trained language models have been tremendously successful in most of the conventional NLP tasks. But they often struggle in those tasks where numerical understanding is required. Some possible reasons can be the tokenizers and pre-training objectives which are not specifically designed to learn and preserve numeracy. Here we investigate the ability of text-to-text transfer learning model (T5), which has outperformed its predecessors in the conventional NLP tasks, to learn numeracy. We consider four numeracy tasks: numeration, magnitude order prediction, finding minimum and maximum in a series, and sorting. We find that, although T5 models perform reasonably well in the interpolation setting, they struggle considerably in the extrapolation setting across all four tasks.