Ghost Policies: A New Paradigm for Understanding and Learning from Failure in Deep Reinforcement Learning

TL;DR

This paper introduces Ghost Policies and the Arvolution AR framework to visualize and learn from DRL agent failures, aiming to improve understanding, debugging, and robustness of deep reinforcement learning agents.

Contribution

It presents a novel AR-based visualization method for DRL failures and a systematic approach for studying and learning from these failures, pioneering a new research paradigm.

Findings

AR visualization of ghost policies enhances failure interpretability

Systematic human disruption improves failure analysis

Dual learning loop facilitates mutual learning from failures

Abstract

Deep Reinforcement Learning (DRL) agents often exhibit intricate failure modes that are difficult to understand, debug, and learn from. This opacity hinders their reliable deployment in real-world applications. To address this critical gap, we introduce ``Ghost Policies,'' a concept materialized through Arvolution, a novel Augmented Reality (AR) framework. Arvolution renders an agent's historical failed policy trajectories as semi-transparent ``ghosts'' that coexist spatially and temporally with the active agent, enabling an intuitive visualization of policy divergence. Arvolution uniquely integrates: (1) AR visualization of ghost policies, (2) a behavioural taxonomy of DRL maladaptation, (3) a protocol for systematic human disruption to scientifically study failure, and (4) a dual-learning loop where both humans and agents learn from these visualized failures. We propose a paradigm shift, transforming DRL agent failures from opaque, costly errors into invaluable, actionable learning resources, laying the groundwork for a new research field: ``Failure Visualization Learning.''