HRM-Agent: Training a recurrent reasoning model in dynamic environments using reinforcement learning

TL;DR

This paper introduces HRM-Agent, a reinforcement learning-trained variant of the Hierarchical Reasoning Model, capable of dynamic environment navigation and reusing computation over time, addressing limitations in previous static applications.

Contribution

It presents HRM-Agent, enabling HRM to operate in dynamic, uncertain environments using reinforcement learning, and demonstrates its ability to reuse previous computation effectively.

Findings

HRM-Agent successfully navigates dynamic maze environments.

Recurrent inference process reuses computation from earlier steps.

HRM's reasoning abilities extend to partially observable, uncertain scenarios.

Abstract

The Hierarchical Reasoning Model (HRM) has impressive reasoning abilities given its small size, but has only been applied to supervised, static, fully-observable problems. One of HRM's strengths is its ability to adapt its computational effort to the difficulty of the problem. However, in its current form it cannot integrate and reuse computation from previous time-steps if the problem is dynamic, uncertain or partially observable, or be applied where the correct action is undefined, characteristics of many real-world problems. This paper presents HRM-Agent, a variant of HRM trained using only reinforcement learning. We show that HRM can learn to navigate to goals in dynamic and uncertain maze environments. Recent work suggests that HRM's reasoning abilities stem from its recurrent inference process. We explore the dynamics of the recurrent inference process and find evidence that it is successfully reusing computation from earlier environment time-steps.