Do Language Models Understand Measurements?

TL;DR

This paper investigates whether pre-trained language models can understand measurements, finds they lack this ability, but can improve with a new embedding strategy and measurement-rich training data.

Contribution

It introduces a simple embedding method to enhance measurement understanding and demonstrates the importance of measurement-rich training data for PLMs.

Findings

PLMs lack inherent measurement reasoning capabilities

Training on measurement-rich corpora improves understanding

Embedding strategies significantly boost measurement comprehension

Abstract

Recent success of pre-trained language models (PLMs) has stimulated interest in their ability to understand and work with numbers. Yet, the numerical reasoning over measurements has not been formally studied despite their importance. In this study, we show that PLMs lack the capability required for reasoning over measurements. Furthermore, we find that a language model trained on a measurement-rich corpus shows better performance on understanding measurements. We propose a simple embedding strategy to better distinguish between numbers and units, which leads to a significant improvement in the probing tasks.