On Reducing Undesirable Behavior in Deep Reinforcement Learning Models

TL;DR

This paper introduces a framework that reduces undesirable behaviors in deep reinforcement learning models by integrating decision tree classifiers into training, with minimal performance impact and improved safety.

Contribution

It presents a novel method that extracts decision trees from errors and incorporates them into training to penalize undesirable behaviors in DRL models.

Findings

Significantly reduces undesirable behaviors in DRL models.

Incur minimal overhead and slight performance impact.

Can even improve overall performance in some cases.

Abstract

Deep reinforcement learning (DRL) has proven extremely useful in a large variety of application domains. However, even successful DRL-based software can exhibit highly undesirable behavior. This is due to DRL training being based on maximizing a reward function, which typically captures general trends but cannot precisely capture, or rule out, certain behaviors of the system. In this paper, we propose a novel framework aimed at drastically reducing the undesirable behavior of DRL-based software, while maintaining its excellent performance. In addition, our framework can assist in providing engineers with a comprehensible characterization of such undesirable behavior. Under the hood, our approach is based on extracting decision tree classifiers from erroneous state-action pairs, and then integrating these trees into the DRL training loop, penalizing the system whenever it performs an error. We provide a proof-of-concept implementation of our approach, and use it to evaluate the technique on three significant case studies. We find that our approach can extend existing frameworks in a straightforward manner, and incurs only a slight overhead in training time. Further, it incurs only a very slight hit to performance, or even in some cases - improves it, while significantly reducing the frequency of undesirable behavior.