Learning neuro-symbolic convergent term rewriting systems

TL;DR

This paper introduces neuro-symbolic architectures, NRS and FastNRS, for learning convergent term rewriting systems that generalize well to out-of-distribution tasks, with FastNRS being more efficient.

Contribution

The paper presents a novel neuro-symbolic framework for learning convergent term rewriting systems, including two modular models that outperform existing neural baselines.

Findings

Models generalize to out-of-distribution instances.

FastNRS improves memory efficiency and training speed.

System outperforms strong neural baselines and matches or exceeds GPT-4o.

Abstract

Building neural systems that can learn to execute symbolic algorithms is a challenging open problem in artificial intelligence, especially when aiming for strong generalization and out-of-distribution performance. In this work, we introduce a general framework for learning convergent term rewriting systems using a neuro-symbolic architecture inspired by the rewriting algorithm itself. We present two modular implementations of such architecture: the Neural Rewriting System (NRS) and the Fast Neural Rewriting System (FastNRS). As a result of algorithmic-inspired design and key architectural elements, both models can generalize to out-of-distribution instances, with FastNRS offering significant improvements in terms of memory efficiency, training speed, and inference time. We evaluate both architectures on four tasks involving the simplification of mathematical formulas and further demonstrate their versatility in a multi-domain learning scenario, where a single model is trained to solve multiple types of problems simultaneously. The proposed system significantly outperforms two strong neural baselines: the Neural Data Router, a recent transformer variant specifically designed to solve algorithmic problems, and GPT-4o, one of the most powerful general-purpose large-language models. Moreover, our system matches or outperforms the latest o1-preview model from OpenAI that excels in reasoning benchmarks.