VASE: Variational Assorted Surprise Exploration for Reinforcement Learning

TL;DR

VASE introduces a novel surprise-based exploration method using Bayesian neural networks and variational inference, significantly improving exploration efficiency in continuous control RL tasks with sparse rewards.

Contribution

It proposes a new definition of surprise and implements it with Bayesian neural networks trained via variational inference for enhanced exploration in RL.

Findings

VASE outperforms existing surprise-based methods in continuous control environments.

The approach effectively balances model accuracy and policy updates.

Experimental results demonstrate superior exploration efficiency with VASE.

Abstract

Exploration in environments with continuous control and sparse rewards remains a key challenge in reinforcement learning (RL). Recently, surprise has been used as an intrinsic reward that encourages systematic and efficient exploration. We introduce a new definition of surprise and its RL implementation named Variational Assorted Surprise Exploration (VASE). VASE uses a Bayesian neural network as a model of the environment dynamics and is trained using variational inference, alternately updating the accuracy of the agent's model and policy. Our experiments show that in continuous control sparse reward environments VASE outperforms other surprise-based exploration techniques.