Experimental Evidence that Empowerment May Drive Exploration in Sparse-Reward Environments

TL;DR

This paper provides experimental evidence that empowerment-based intrinsic motivation can effectively promote exploration in sparse-reward reinforcement learning environments, comparable to curiosity-driven methods.

Contribution

It introduces an empowerment-inspired agent and compares it with a curiosity-based agent, demonstrating empowerment's potential in driving exploration.

Findings

Both agents benefit similarly from intrinsic rewards.

Empowerment can be an effective exploration strategy.

Experimental results support empowerment's role in sparse environments.

Abstract

Reinforcement Learning (RL) is known to be often unsuccessful in environments with sparse extrinsic rewards. A possible countermeasure is to endow RL agents with an intrinsic reward function, or 'intrinsic motivation', which rewards the agent based on certain features of the current sensor state. An intrinsic reward function based on the principle of empowerment assigns rewards proportional to the amount of control the agent has over its own sensors. We implemented a variation on a recently proposed intrinsically motivated agent, which we refer to as the 'curious' agent, and an empowerment-inspired agent. The former leverages sensor state encoding with a variational autoencoder, while the latter predicts the next sensor state via a variational information bottleneck. We compared the performance of both agents to that of an advantage actor-critic baseline in four sparse reward grid worlds. Both the empowerment agent and its curious competitor seem to benefit to similar extents from their intrinsic rewards. This provides some experimental support to the conjecture that empowerment can be used to drive exploration.