Interpretable Reinforcement Learning with Multilevel Subgoal Discovery

TL;DR

This paper introduces an interpretable reinforcement learning model that discovers hierarchical subgoals in discrete environments without requiring reward functions, using probabilistic rules and state descriptions for efficient policy learning.

Contribution

It presents a novel RL framework that learns environment rules and subgoal hierarchies in an interpretable manner without reward signals.

Findings

Supports hierarchical subgoal discovery

Enables interpretable policy learning

Improves efficiency through state-based subgoals

Abstract

We propose a novel Reinforcement Learning model for discrete environments, which is inherently interpretable and supports the discovery of deep subgoal hierarchies. In the model, an agent learns information about environment in the form of probabilistic rules, while policies for (sub)goals are learned as combinations thereof. No reward function is required for learning; an agent only needs to be given a primary goal to achieve. Subgoals of a goal G from the hierarchy are computed as descriptions of states, which if previously achieved increase the total efficiency of the available policies for G. These state descriptions are introduced as new sensor predicates into the rule language of the agent, which allows for sensing important intermediate states and for updating environment rules and policies accordingly.