GRAIL: Autonomous Concept Grounding for Neuro-Symbolic Reinforcement Learning

TL;DR

GRAIL introduces an autonomous framework that uses large language models and interactive learning to ground relational concepts in reinforcement learning environments, enhancing interpretability and adaptability.

Contribution

It automates the grounding of relational concepts in neuro-symbolic RL using LLMs and environmental interaction, reducing reliance on manual definitions.

Findings

GRAIL matches or outperforms manually crafted concept agents in Atari games.

Addresses sparse rewards and concept misalignment in complex environments.

Reveals trade-offs between reward maximization and goal completion.

Abstract

Neuro-symbolic Reinforcement Learning (NeSy-RL) combines symbolic reasoning with gradient-based optimization to achieve interpretable and generalizable policies. Relational concepts, such as "left of" or "close by", serve as foundational building blocks that structure how agents perceive and act. However, conventional approaches require human experts to manually define these concepts, limiting adaptability since concept semantics vary across environments. We propose GRAIL (Grounding Relational Agents through Interactive Learning), a framework that autonomously grounds relational concepts through environmental interaction. GRAIL leverages large language models (LLMs) to provide generic concept representations as weak supervision, then refines them to capture environment-specific semantics. This approach addresses both sparse reward signals and concept misalignment prevalent in underdetermined environments. Experiments on the Atari games Kangaroo, Seaquest, and Skiing demonstrate that GRAIL matches or outperforms agents with manually crafted concepts in simplified settings, and reveals informative trade-offs between reward maximization and high-level goal completion in the full environment.