An Intentional Forgetting-Driven Self-Healing Method For Deep Reinforcement Learning Systems

TL;DR

This paper introduces Dr. DRL, a self-healing method for deep reinforcement learning that incorporates intentional forgetting to improve adaptation speed and success rate in environments with significant shifts.

Contribution

It proposes a novel intentional forgetting mechanism integrated into continual learning to address environmental drifts in DRL systems, enhancing adaptation and performance.

Findings

Reduces healing time by 18.74%

Decreases fine-tuning episodes by 17.72%

Improves adaptation to previously unsolvable drifts by 19.63%

Abstract

Deep reinforcement learning (DRL) is increasingly applied in large-scale productions like Netflix and Facebook. As with most data-driven systems, DRL systems can exhibit undesirable behaviors due to environmental drifts, which often occur in constantly-changing production settings. Continual Learning (CL) is the inherent self-healing approach for adapting the DRL agent in response to the environment's conditions shifts. However, successive shifts of considerable magnitude may cause the production environment to drift from its original state. Recent studies have shown that these environmental drifts tend to drive CL into long, or even unsuccessful, healing cycles, which arise from inefficiencies such as catastrophic forgetting, warm-starting failure, and slow convergence. In this paper, we propose Dr. DRL, an effective self-healing approach for DRL systems that integrates a novel mechanism of intentional forgetting into vanilla CL to overcome its main issues. Dr. DRL deliberately erases the DRL system's minor behaviors to systematically prioritize the adaptation of the key problem-solving skills. Using well-established DRL algorithms, Dr. DRL is compared with vanilla CL on various drifted environments. Dr. DRL is able to reduce, on average, the healing time and fine-tuning episodes by, respectively, 18.74% and 17.72%. Dr. DRL successfully helps agents to adapt to 19.63% of drifted environments left unsolved by vanilla CL while maintaining and even enhancing by up to 45% the obtained rewards for drifted environments that are resolved by both approaches.