Can neural networks do arithmetic? A survey on the elementary numerical skills of state-of-the-art deep learning models

TL;DR

This survey critically examines whether current deep learning models truly understand basic numerical and arithmetic concepts, highlighting their limitations despite recent successes in mathematical problem-solving tasks.

Contribution

It provides a comprehensive review of recent neural network architectures and benchmarks, revealing gaps in elementary numerical understanding.

Findings

State-of-the-art models often fail basic arithmetic tasks

Neural networks lack elementary numerical reasoning skills

Current models do not fully grasp symbolic quantities

Abstract

Creating learning models that can exhibit sophisticated reasoning skills is one of the greatest challenges in deep learning research, and mathematics is rapidly becoming one of the target domains for assessing scientific progress in this direction. In the past few years there has been an explosion of neural network architectures, data sets, and benchmarks specifically designed to tackle mathematical problems, reporting notable success in disparate fields such as automated theorem proving, numerical integration, and discovery of new conjectures or matrix multiplication algorithms. However, despite these impressive achievements it is still unclear whether deep learning models possess an elementary understanding of quantities and symbolic numbers. In this survey we critically examine the recent literature, concluding that even state-of-the-art architectures often fall short when probed with relatively simple tasks designed to test basic numerical and arithmetic knowledge.