From Passive Reuse to Active Reasoning: Grounding Large Language Models for Neuro-Symbolic Experience Replay

TL;DR

This paper introduces Neuro-Symbolic Experience Replay (NSER), a framework that uses large language models to actively extract and ground behavioral rules from experiences, improving reinforcement learning efficiency.

Contribution

It presents a novel neuro-symbolic grounding pipeline that enables LLMs to induce and ground behavioral rules, transforming experience replay into an active knowledge construction process.

Findings

NSER improves sample efficiency across various benchmarks.

Grounded symbolic rules enhance policy convergence speed.

The framework effectively integrates linguistic reasoning with RL.

Abstract

While experience replay is essential for data efficiency in reinforcement learning (RL), standard methods treat the replay buffer as a passive memory system, prioritizing samples based on numerical prediction errors rather than their semantic significance. This approach stands in contrast to human learning, which accelerates mastery by actively abstracting fragmented experiences into behavioral rules. To bridge this gap, we propose Neuro-Symbolic Experience Replay (NSER), a framework that transforms experience replay from a passive sample reuse mechanism into an active engine for knowledge construction. Specifically, NSER addresses the incompatibility between linguistic reasoning and numerical optimization through a novel neuro-symbolic grounding pipeline. It leverages Large Language Models (LLMs) in a zero-shot manner to induce candidate behavioral rules from accumulated trajectories, grounds these insights into differentiable first-order logic representations, and utilizes the resulting symbolic structures to dynamically reweight the replay distribution. By allowing abstract knowledge to directly shape policy optimization, NSER achieves consistent superior sample efficiency and convergence speed across reactive, rule-based, and procedural benchmarks.