A User Study on Explainable Online Reinforcement Learning for Adaptive Systems

TL;DR

This paper presents an empirical user study evaluating the effectiveness and perceived usefulness of an explainable deep reinforcement learning technique, XRL-DINE, in aiding software engineers to understand and debug adaptive systems.

Contribution

It introduces an empirical evaluation of XRL-DINE, an explainable RL method, assessing its impact on software engineers' performance and usability perceptions.

Findings

Software engineers performed better with XRL-DINE.

Participants found XRL-DINE useful for understanding decisions.

Ease of use was positively rated by users.

Abstract

Online reinforcement learning (RL) is increasingly used for realizing adaptive systems in the presence of design time uncertainty. Online RL facilitates learning from actual operational data and thereby leverages feedback only available at runtime. However, Online RL requires the definition of an effective and correct reward function, which quantifies the feedback to the RL algorithm and thereby guides learning. With Deep RL gaining interest, the learned knowledge is no longer explicitly represented, but is represented as a neural network. For a human, it becomes practically impossible to relate the parametrization of the neural network to concrete RL decisions. Deep RL thus essentially appears as a black box, which severely limits the debugging of adaptive systems. We previously introduced the explainable RL technique XRL-DINE, which provides visual insights into why certain decisions were made at important time points. Here, we introduce an empirical user study involving 54 software engineers from academia and industry to assess (1) the performance of software engineers when performing different tasks using XRL-DINE and (2) the perceived usefulness and ease of use of XRL-DINE.