Probing for Multilingual Numerical Understanding in Transformer-Based Language Models

TL;DR

This paper investigates whether multilingual transformer models understand numerical compositionality in language by testing their ability to perform grammaticality and value comparison tasks across several languages.

Contribution

It introduces novel multilingual probing tasks to evaluate the models' understanding of numerical structures and compositional reasoning in natural language numbers.

Findings

Models encode enough information for grammaticality judgments.

Models generally lack sufficient information for accurate value comparisons.

Performance varies across languages and tasks.

Abstract

Natural language numbers are an example of compositional structures, where larger numbers are composed of operations on smaller numbers. Given that compositional reasoning is a key to natural language understanding, we propose novel multilingual probing tasks tested on DistilBERT, XLM, and BERT to investigate for evidence of compositional reasoning over numerical data in various natural language number systems. By using both grammaticality judgment and value comparison classification tasks in English, Japanese, Danish, and French, we find evidence that the information encoded in these pretrained models' embeddings is sufficient for grammaticality judgments but generally not for value comparisons. We analyze possible reasons for this and discuss how our tasks could be extended in further studies.