Satisficing Exploration for Deep Reinforcement Learning

TL;DR

This paper introduces a deep reinforcement learning approach that enables agents to efficiently learn satisficing policies by directly representing uncertainty over the value function, bypassing model-based planning.

Contribution

It extends existing satisficing exploration methods to deep RL by removing the need for model-based planning, allowing for efficient learning in high-dimensional environments.

Findings

Enables deep RL agents to learn satisficing behaviors effectively.

Achieves more efficient synthesis of optimal behaviors when feasible.

Demonstrates the approach with simple experiments.

Abstract

A default assumption in the design of reinforcement-learning algorithms is that a decision-making agent always explores to learn optimal behavior. In sufficiently complex environments that approach the vastness and scale of the real world, however, attaining optimal performance may in fact be an entirely intractable endeavor and an agent may seldom find itself in a position to complete the requisite exploration for identifying an optimal policy. Recent work has leveraged tools from information theory to design agents that deliberately forgo optimal solutions in favor of sufficiently-satisfying or satisficing solutions, obtained through lossy compression. Notably, such agents may employ fundamentally different exploratory decisions to learn satisficing behaviors more efficiently than optimal ones that are more data intensive. While supported by a rigorous corroborating theory, the underlying algorithm relies on model-based planning, drastically limiting the compatibility of these ideas with function approximation and high-dimensional observations. In this work, we remedy this issue by extending an agent that directly represents uncertainty over the optimal value function allowing it to both bypass the need for model-based planning and to learn satisficing policies. We provide simple yet illustrative experiments that demonstrate how our algorithm enables deep reinforcement-learning agents to achieve satisficing behaviors. In keeping with previous work on this setting for multi-armed bandits, we additionally find that our algorithm is capable of synthesizing optimal behaviors, when feasible, more efficiently than its non-information-theoretic counterpart.