Sample-Efficient Neurosymbolic Deep Reinforcement Learning

TL;DR

This paper introduces a neuro-symbolic deep reinforcement learning method that leverages symbolic knowledge and logical rules to improve sample efficiency, generalization, and interpretability in complex decision-making tasks.

Contribution

It presents a novel integration of symbolic reasoning with deep RL, using logical rules to guide exploration and exploitation, enhancing performance and trustworthiness.

Findings

Achieves better sample efficiency than baseline methods.

Improves generalization to unseen, complex tasks.

Outperforms reward machine baseline in gridworld environments.

Abstract

Reinforcement Learning (RL) is a well-established framework for sequential decision-making in complex environments. However, state-of-the-art Deep RL (DRL) algorithms typically require large training datasets and often struggle to generalize beyond small-scale training scenarios, even within standard benchmarks. We propose a neuro-symbolic DRL approach that integrates background symbolic knowledge to improve sample efficiency and generalization to more challenging, unseen tasks. Partial policies defined for simple domain instances, where high performance is easily attained, are transferred as useful priors to accelerate learning in more complex settings and avoid tuning DRL parameters from scratch. To do so, partial policies are represented as logical rules, and online reasoning is performed to guide the training process through two mechanisms: (i) biasing the action distribution during exploration, and (ii) rescaling Q-values during exploitation. This neuro-symbolic integration enhances interpretability and trustworthiness while accelerating convergence, particularly in sparse-reward environments and tasks with long planning horizons. We empirically validate our methodology on challenging variants of gridworld environments, both in the fully observable and partially observable setting. We show improved performance over a state-of-the-art reward machine baseline.