GalilAI: Out-of-Task Distribution Detection using Causal Active Experimentation for Safe Transfer RL

TL;DR

GalilAI introduces a causal active experimentation approach for out-of-task distribution detection in reinforcement learning, enabling agents to identify environment shifts through active testing, thus improving robustness and safety.

Contribution

The paper proposes a novel causal framework and an active experimentation method, GalilAI, for out-of-task distribution detection in RL, addressing a key gap in safe transfer learning.

Findings

GalilAI outperforms the baseline in OOTD detection accuracy.

Active experimentation improves environment shift detection.

The causal framework guides effective exploration for OOD detection.

Abstract

Out-of-distribution (OOD) detection is a well-studied topic in supervised learning. Extending the successes in supervised learning methods to the reinforcement learning (RL) setting, however, is difficult due to the data generating process - RL agents actively query their environment for data, and the data are a function of the policy followed by the agent. An agent could thus neglect a shift in the environment if its policy did not lead it to explore the aspect of the environment that shifted. Therefore, to achieve safe and robust generalization in RL, there exists an unmet need for OOD detection through active experimentation. Here, we attempt to bridge this lacuna by first defining a causal framework for OOD scenarios or environments encountered by RL agents in the wild. Then, we propose a novel task: that of Out-of-Task Distribution (OOTD) detection. We introduce an RL agent that actively experiments in a test environment and subsequently concludes whether it is OOTD or not. We name our method GalilAI, in honor of Galileo Galilei, as it discovers, among other causal processes, that gravitational acceleration is independent of the mass of a body. Finally, we propose a simple probabilistic neural network baseline for comparison, which extends extant Model-Based RL. We find that GalilAI outperforms the baseline significantly. See visualizations of our method https://galil-ai.github.io/