MIRA: Memory-Integrated Reinforcement Learning Agent with Limited LLM Guidance

TL;DR

MIRA introduces a memory-augmented reinforcement learning framework that leverages structured memory graphs to reduce reliance on continuous LLM supervision, improving early learning in sparse reward environments.

Contribution

MIRA integrates a structured memory graph with RL to amortize LLM queries, enhancing early learning without continuous supervision and providing theoretical guarantees.

Findings

MIRA outperforms baseline RL methods in sparse reward tasks.

MIRA achieves comparable performance to LLM-supervised approaches with fewer LLM queries.

The utility-based shaping accelerates early-stage learning in sparse environments.

Abstract

Reinforcement learning (RL) agents often suffer from high sample complexity in sparse or delayed reward settings due to limited prior structure. Large language models (LLMs) can provide subgoal decompositions, plausible trajectories, and abstract priors that facilitate early learning. However, heavy reliance on LLM supervision introduces scalability constraints and dependence on potentially unreliable signals. We propose MIRA (Memory-Integrated Reinforcement Learning Agent), which incorporates a structured, evolving memory graph to guide early training. The graph stores decision-relevant information, including trajectory segments and subgoal structures, and is constructed from both the agent's high-return experiences and LLM outputs. This design amortizes LLM queries into a persistent memory rather than requiring continuous real-time supervision. From this memory graph, we derive a utility signal that softly adjusts advantage estimation to influence policy updates without modifying the underlying reward function. As training progresses, the agent's policy gradually surpasses the initial LLM-derived priors, and the utility term decays, preserving standard convergence guarantees. We provide theoretical analysis showing that utility-based shaping improves early-stage learning in sparse-reward environments. Empirically, MIRA outperforms RL baselines and achieves returns comparable to approaches that rely on frequent LLM supervision, while requiring substantially fewer online LLM queries. Project webpage: https://narjesno.github.io/MIRA/