Value of Information-Enhanced Exploration in Bootstrapped DQN

TL;DR

This paper enhances deep exploration in reinforcement learning by integrating value of information into Bootstrapped DQN, leading to improved performance in complex, sparse-reward environments without extra hyperparameters.

Contribution

The paper introduces two novel algorithms that incorporate value of information estimates into Bootstrapped DQN to improve exploration efficiency.

Findings

Enhanced performance in Atari games with sparse rewards.

Better utilization of uncertainty without additional hyperparameters.

Improved exploration compared to traditional methods.

Abstract

Efficient exploration in deep reinforcement learning remains a fundamental challenge, especially in environments characterized by high-dimensional states and sparse rewards. Traditional exploration strategies that rely on random local policy noise, such as $\epsilon$-greedy and Boltzmann exploration methods, often struggle to efficiently balance exploration and exploitation. In this paper, we integrate the notion of (expected) value of information (EVOI) within the well-known Bootstrapped DQN algorithmic framework, to enhance the algorithm's deep exploration ability. Specifically, we develop two novel algorithms that incorporate the expected gain from learning the value of information into Bootstrapped DQN. Our methods use value of information estimates to measure the discrepancies of opinions among distinct network heads, and drive exploration towards areas with the most potential. We evaluate our algorithms with respect to performance and their ability to exploit inherent uncertainty arising from random network initialization. Our experiments in complex, sparse-reward Atari games demonstrate increased performance, all the while making better use of uncertainty, and, importantly, without introducing extra hyperparameters.