Online inductive learning from answer sets for efficient reinforcement learning exploration

TL;DR

This paper introduces an online inductive logic programming method integrated with reinforcement learning to enhance training efficiency and explainability, demonstrated through improved performance in a Pac-Man scenario.

Contribution

It presents a novel online approach combining answer set learning with RL to guide exploration without reward shaping, maintaining optimality and explainability.

Findings

Significant boost in discounted return during early training batches

Inductive rules quickly converge to explain the agent policy

Method maintains computational efficiency of Q-learning

Abstract

This paper presents a novel approach combining inductive logic programming with reinforcement learning to improve training performance and explainability. We exploit inductive learning of answer set programs from noisy examples to learn a set of logical rules representing an explainable approximation of the agent policy at each batch of experience. We then perform answer set reasoning on the learned rules to guide the exploration of the learning agent at the next batch, without requiring inefficient reward shaping and preserving optimality with soft bias. The entire procedure is conducted during the online execution of the reinforcement learning algorithm. We preliminarily validate the efficacy of our approach by integrating it into the Q-learning algorithm for the Pac-Man scenario in two maps of increasing complexity. Our methodology produces a significant boost in the discounted return achieved by the agent, even in the first batches of training. Moreover, inductive learning does not compromise the computational time required by Q-learning and learned rules quickly converge to an explanation of the agent policy.