Beyond Interpolation: Extrapolative Reasoning with Reinforcement Learning and Graph Neural Networks

TL;DR

This paper explores how graph neural networks combined with reinforcement learning can improve the ability of models to extrapolate and reason correctly on complex logic puzzles beyond their training data, emphasizing architectural biases and reward design.

Contribution

It introduces a graph-based reinforcement learning framework for logical reasoning that enhances generalization to larger, more complex puzzles, highlighting the importance of inductive biases and recurrent modeling.

Findings

Graph-based models improve extrapolation on logic puzzles

Recurrent modeling enhances sequential reasoning capabilities

Reward system design significantly impacts generalization performance

Abstract

Despite incredible progress, many neural architectures fail to properly generalize beyond their training distribution. As such, learning to reason in a correct and generalizable way is one of the current fundamental challenges in machine learning. In this respect, logic puzzles provide a great testbed, as we can fully understand and control the learning environment. Thus, they allow to evaluate performance on previously unseen, larger and more difficult puzzles that follow the same underlying rules. Since traditional approaches often struggle to represent such scalable logical structures, we propose to model these puzzles using a graph-based approach. Then, we investigate the key factors enabling the proposed models to learn generalizable solutions in a reinforcement learning setting. Our study focuses on the impact of the inductive bias of the architecture, different reward systems and the role of recurrent modeling in enabling sequential reasoning. Through extensive experiments, we demonstrate how these elements contribute to successful extrapolation on increasingly complex puzzles.These insights and frameworks offer a systematic way to design learning-based systems capable of generalizable reasoning beyond interpolation.