Neural Sequence-to-grid Module for Learning Symbolic Rules

TL;DR

This paper introduces a neural sequence-to-grid module that preprocesses inputs into a grid format, enabling neural networks to generalize better on symbolic reasoning tasks, including arithmetic, algorithms, and QA, especially out-of-distribution.

Contribution

The paper presents a novel differentiable sequence-to-grid module that improves neural networks' out-of-distribution generalization on symbolic reasoning tasks.

Findings

Achieves OOD generalization on arithmetic and algorithmic problems

Enhances TextCNN to solve bAbI QA tasks without external memory

Outperforms existing sequence transduction models

Abstract

Logical reasoning tasks over symbols, such as learning arithmetic operations and computer program evaluations, have become challenges to deep learning. In particular, even state-of-the-art neural networks fail to achieve \textit{out-of-distribution} (OOD) generalization of symbolic reasoning tasks, whereas humans can easily extend learned symbolic rules. To resolve this difficulty, we propose a neural sequence-to-grid (seq2grid) module, an input preprocessor that automatically segments and aligns an input sequence into a grid. As our module outputs a grid via a novel differentiable mapping, any neural network structure taking a grid input, such as ResNet or TextCNN, can be jointly trained with our module in an end-to-end fashion. Extensive experiments show that neural networks having our module as an input preprocessor achieve OOD generalization on various arithmetic and algorithmic problems including number sequence prediction problems, algebraic word problems, and computer program evaluation problems while other state-of-the-art sequence transduction models cannot. Moreover, we verify that our module enhances TextCNN to solve the bAbI QA tasks without external memory.