Interpretable Option Discovery using Deep Q-Learning and Variational Autoencoders

TL;DR

This paper introduces the Deep Variational Q-Network (DVQN), a novel method combining deep generative models and reinforcement learning to automatically discover options with interpretable initiation and termination conditions, improving sample efficiency and stability.

Contribution

The paper presents DVQN, a new algorithm that automates option discovery in deep RL using a variational autoencoder framework, enhancing interpretability and performance.

Findings

DVQN achieves comparable performance to Rainbow.

DVQN maintains stability during extended training.

Automatic initiation and termination improve option learning.

Abstract

Deep Reinforcement Learning (RL) is unquestionably a robust framework to train autonomous agents in a wide variety of disciplines. However, traditional deep and shallow model-free RL algorithms suffer from low sample efficiency and inadequate generalization for sparse state spaces. The options framework with temporal abstractions is perhaps the most promising method to solve these problems, but it still has noticeable shortcomings. It only guarantees local convergence, and it is challenging to automate initiation and termination conditions, which in practice are commonly hand-crafted. Our proposal, the Deep Variational Q-Network (DVQN), combines deep generative- and reinforcement learning. The algorithm finds good policies from a Gaussian distributed latent-space, which is especially useful for defining options. The DVQN algorithm uses MSE with KL-divergence as regularization, combined with traditional Q-Learning updates. The algorithm learns a latent-space that represents good policies with state clusters for options. We show that the DVQN algorithm is a promising approach for identifying initiation and termination conditions for option-based reinforcement learning. Experiments show that the DVQN algorithm, with automatic initiation and termination, has comparable performance to Rainbow and can maintain stability when trained for extended periods after convergence.