Confounding Robust Continuous Control via Automatic Reward Shaping

TL;DR

This paper introduces a method to automatically learn reward shaping functions for continuous control in reinforcement learning, robust to unobserved confounders, using causal inference techniques and tested with strong results on benchmark tasks.

Contribution

It presents a novel approach to automatically learn confounding-robust reward shaping functions from offline data using causal Bellman equations and potential-based reward shaping.

Findings

Strong performance on continuous control benchmarks

Robustness to unobserved confounders demonstrated

First causal perspective approach in this context

Abstract

Reward shaping has been applied widely to accelerate Reinforcement Learning (RL) agents' training. However, a principled way of designing effective reward shaping functions, especially for complex continuous control problems, remains largely under-explained. In this work, we propose to automatically learn a reward shaping function for continuous control problems from offline datasets, potentially contaminated by unobserved confounding variables. Specifically, our method builds upon the recently proposed causal Bellman equation to learn a tight upper bound on the optimal state values, which is then used as the potentials in the Potential-Based Reward Shaping (PBRS) framework. Our proposed reward shaping algorithm is tested with Soft-Actor-Critic (SAC) on multiple commonly used continuous control benchmarks and exhibits strong performance guarantees under unobserved confounders. More broadly, our work marks a solid first step towards confounding robust continuous control from a causal perspective. Code for training our reward shaping functions can be found at https://github.com/mateojuliani/confounding_robust_cont_control.