A Primer for Neural Arithmetic Logic Modules

TL;DR

This paper reviews the progress of Neural Arithmetic Logic Modules, analyzes key designs like NALU, identifies inconsistencies in experiments, and introduces a benchmark for fair comparison, guiding future research.

Contribution

It provides the first comprehensive overview of the field, analyzes design choices, highlights experimental inconsistencies, and introduces a benchmark for neural arithmetic modules.

Findings

Inconsistencies exist in experimental setups across studies.

A new benchmark enables fair comparison of NALMs.

Analysis of NALU design choices informs future module development.

Abstract

Neural Arithmetic Logic Modules have become a growing area of interest, though remain a niche field. These modules are neural networks which aim to achieve systematic generalisation in learning arithmetic and/or logic operations such as $\{+, -, \times, \div, \leq, \textrm{AND}\}$ while also being interpretable. This paper is the first in discussing the current state of progress of this field, explaining key works, starting with the Neural Arithmetic Logic Unit (NALU). Focusing on the shortcomings of the NALU, we provide an in-depth analysis to reason about design choices of recent modules. A cross-comparison between modules is made on experiment setups and findings, where we highlight inconsistencies in a fundamental experiment causing the inability to directly compare across papers. To alleviate the existing inconsistencies, we create a benchmark which compares all existing arithmetic NALMs. We finish by providing a novel discussion of existing applications for NALU and research directions requiring further exploration.