Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlation

TL;DR

This paper introduces RSC-MDPs, a new framework for robust reinforcement learning that effectively avoids learning spurious correlations caused by unobserved confounders, demonstrated through theoretical analysis and superior empirical performance.

Contribution

We propose RSC-MDPs to address robustness against spurious correlations in RL and develop an empirical algorithm that outperforms existing methods in real-world tasks.

Findings

RSC-MDPs outperform existing robust RL methods in avoiding spurious correlations.

The empirical algorithm achieves superior results in self-driving and manipulation tasks.

Theoretical analysis confirms the superiority of RSC-MDPs in confounder scenarios.

Abstract

Robustness has been extensively studied in reinforcement learning (RL) to handle various forms of uncertainty such as random perturbations, rare events, and malicious attacks. In this work, we consider one critical type of robustness against spurious correlation, where different portions of the state do not have correlations induced by unobserved confounders. These spurious correlations are ubiquitous in real-world tasks, for instance, a self-driving car usually observes heavy traffic in the daytime and light traffic at night due to unobservable human activity. A model that learns such useless or even harmful correlation could catastrophically fail when the confounder in the test case deviates from the training one. Although motivated, enabling robustness against spurious correlation poses significant challenges since the uncertainty set, shaped by the unobserved confounder and causal structure, is difficult to characterize and identify. Existing robust algorithms that assume simple and unstructured uncertainty sets are therefore inadequate to address this challenge. To solve this issue, we propose Robust State-Confounded Markov Decision Processes (RSC-MDPs) and theoretically demonstrate its superiority in avoiding learning spurious correlations compared with other robust RL counterparts. We also design an empirical algorithm to learn the robust optimal policy for RSC-MDPs, which outperforms all baselines in eight realistic self-driving and manipulation tasks.