Learning Rules from Rewards

TL;DR

This paper introduces RRTL, a relational reinforcement learning model that learns task-specific rules from rewards in complex Atari games, demonstrating how structured relational knowledge guides adaptive behavior.

Contribution

The paper presents RRTL, a novel model that incrementally learns rules from rewards using relational inputs, diverging from traditional methods by focusing on ground rules for specific object configurations.

Findings

RRTL effectively learns relevant relations in Atari games of increasing complexity.

Partitioning state space with relative magnitude values improves learning robustness.

Relational signals can be guided by reinforcement feedback to learn structured representations.

Abstract

Humans can flexibly generalize knowledge across domains by leveraging structured relational representations. While prior research has shown how such representations support analogical reasoning, less is known about how they are recruited to guide adaptive behavior. We address this gap by introducing the Relational Regression Tree Learner (RRTL), a model that incrementally builds policies over structured relational inputs by selecting task-relevant relations during the learning process. RRTL is grounded in the framework of relational reinforcement learning but diverges from traditional approaches by focusing on ground (i.e., non-variabilized) rules that refer to specific object configurations. Across three Atari games of increasing relational complexity (Breakout, Pong, Demon Attack), the model learns to act effectively by identifying a small set of relevant relations from a broad pool of candidate relations. A comparative version of the model, which partitions the state space using relative magnitude values (e.g., "more", "same", "less"), showed more robust learning than a version using logical (binary) splits. These results provide a proof of principle that reinforcement signals can guide the selection of structured representations, offering a computational framework for understanding how relational knowledge is learned and deployed in adaptive behavior.