Open-Book Neural Algorithmic Reasoning

TL;DR

This paper introduces an open-book neural reasoning framework allowing models to access all training instances during reasoning, significantly improving performance on complex algorithmic tasks and revealing task relationships.

Contribution

It proposes a novel open-book learning paradigm for neural algorithmic reasoning and demonstrates its effectiveness on the CLRS benchmark, including multi-task insights.

Findings

Enhanced reasoning capabilities on CLRS benchmark

Open-book attention reveals task relationships

Improved multi-task training interpretability

Abstract

Neural algorithmic reasoning is an emerging area of machine learning that focuses on building neural networks capable of solving complex algorithmic tasks. Recent advancements predominantly follow the standard supervised learning paradigm -- feeding an individual problem instance into the network each time and training it to approximate the execution steps of a classical algorithm. We challenge this mode and propose a novel open-book learning framework. In this framework, whether during training or testing, the network can access and utilize all instances in the training dataset when reasoning for a given instance. Empirical evaluation is conducted on the challenging CLRS Algorithmic Reasoning Benchmark, which consists of 30 diverse algorithmic tasks. Our open-book learning framework exhibits a significant enhancement in neural reasoning capabilities. Further, we notice that there is recent literature suggesting that multi-task training on CLRS can improve the reasoning accuracy of certain tasks, implying intrinsic connections between different algorithmic tasks. We delve into this direction via the open-book framework. When the network reasons for a specific task, we enable it to aggregate information from training instances of other tasks in an attention-based manner. We show that this open-book attention mechanism offers insights into the inherent relationships among various tasks in the benchmark and provides a robust tool for interpretable multi-task training.