Reducing Action Space for Deep Reinforcement Learning via Causal Effect Estimation

TL;DR

This paper introduces a causal effect estimation method to identify and suppress redundant actions in deep reinforcement learning, improving exploration efficiency in large action spaces.

Contribution

It proposes a novel approach combining inverse dynamics modeling and causal effect estimation to quantitatively reduce action redundancy during exploration.

Findings

Enhanced exploration efficiency in environments with large action spaces

Quantitative evidence of action causality improves decision-making

Theoretical analysis supports the method's effectiveness

Abstract

Intelligent decision-making within large and redundant action spaces remains challenging in deep reinforcement learning. Considering similar but ineffective actions at each step can lead to repetitive and unproductive trials. Existing methods attempt to improve agent exploration by reducing or penalizing redundant actions, yet they fail to provide quantitative and reliable evidence to determine redundancy. In this paper, we propose a method to improve exploration efficiency by estimating the causal effects of actions. Unlike prior methods, our approach offers quantitative results regarding the causality of actions for one-step transitions. We first pre-train an inverse dynamics model to serve as prior knowledge of the environment. Subsequently, we classify actions across the entire action space at each time step and estimate the causal effect of each action to suppress redundant actions during exploration. We provide a theoretical analysis to demonstrate the effectiveness of our method and present empirical results from simulations in environments with redundant actions to evaluate its performance. Our implementation is available at https://github.com/agi-brain/cee.git.