Discovering What You Can Control: Interventional Boundary Discovery for Reinforcement Learning

TL;DR

This paper introduces Interventional Boundary Discovery (IBD), a method that uses the agent's actions as interventions to identify controllable observation dimensions in RL, outperforming observational baselines in complex distractor settings.

Contribution

The paper proposes IBD, a novel interventional approach for boundary discovery in RL, effectively distinguishing controllable features amidst distractors.

Findings

IBD matches oracle return in 11 of 12 benchmark settings.

Observational baselines often underperform compared to full observation passing.

IBD effectively identifies controllable dimensions with up to 100 distractors.

Abstract

When an RL agent's observations contain distractors driven by the same confounders as its true state, observational data alone cannot identify which dimensions the agent controls. In our benchmarks, even state-conditioned observational selectors can collapse when distractors mimic controllable state variables. We propose Interventional Boundary Discovery (IBD), which treats the agent's own action channel as a source of randomized interventions: randomizing actions implements an interventional contrast, and per-dimension two-sample tests with FDR correction produce a binary mask over observation dimensions. Across 12 continuous-control settings with up to 100 distractors, IBD matches oracle return in 11 of 12 settings, while observational baselines including mutual information, state-conditioned forward models, and gradient-based sensitivity often underperform simply passing the full observation to SAC.