Toward Causal-Aware RL: State-Wise Action-Refined Temporal Difference

TL;DR

This paper introduces SWAR, a causality-aware reinforcement learning method that intervenes on actions to identify task-relevant actions, improving learning efficiency and interpretability in continuous control tasks.

Contribution

The paper proposes SWAR, a novel causality discovery framework in RL that uses state-wise action refinement and two algorithms, TD-SWAR and Dyn-SWAR, to identify important actions.

Findings

SWAR effectively discovers causal action relationships.

Both algorithms improve learning efficiency in redundant action spaces.

Enhanced interpretability of RL decisions.

Abstract

Although it is well known that exploration plays a key role in Reinforcement Learning (RL), prevailing exploration strategies for continuous control tasks in RL are mainly based on naive isotropic Gaussian noise regardless of the causality relationship between action space and the task and consider all dimensions of actions equally important. In this work, we propose to conduct interventions on the primal action space to discover the causal relationship between the action space and the task reward. We propose the method of State-Wise Action Refined (SWAR), which addresses the issue of action space redundancy and promote causality discovery in RL. We formulate causality discovery in RL tasks as a state-dependent action space selection problem and propose two practical algorithms as solutions. The first approach, TD-SWAR, detects task-related actions during temporal difference learning, while the second approach, Dyn-SWAR, reveals important actions through dynamic model prediction. Empirically, both methods provide approaches to understand the decisions made by RL agents and improve learning efficiency in action-redundant tasks.