Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning

TL;DR

This paper introduces AE-DQN, a deep reinforcement learning architecture that improves learning efficiency by eliminating sub-optimal actions using an external elimination signal, leading to faster and more robust performance in complex environments.

Contribution

The paper presents a novel Action-Elimination Deep Q-Network that integrates an Action Elimination Network to filter out invalid actions, enhancing learning speed and robustness.

Findings

Significant speedup over vanilla DQN in complex environments

Improved robustness in text-based games with many actions

Effective action filtering using external elimination signals

Abstract

Learning how to act when there are many available actions in each state is a challenging task for Reinforcement Learning (RL) agents, especially when many of the actions are redundant or irrelevant. In such cases, it is sometimes easier to learn which actions not to take. In this work, we propose the Action-Elimination Deep Q-Network (AE-DQN) architecture that combines a Deep RL algorithm with an Action Elimination Network (AEN) that eliminates sub-optimal actions. The AEN is trained to predict invalid actions, supervised by an external elimination signal provided by the environment. Simulations demonstrate a considerable speedup and added robustness over vanilla DQN in text-based games with over a thousand discrete actions.