Penalizing side effects using stepwise relative reachability

TL;DR

This paper proposes a novel method for penalizing side effects in reinforcement learning by using stepwise relative reachability, which avoids undesirable incentives present in previous approaches.

Contribution

It introduces a new variant of the stepwise inaction baseline and a relative reachability deviation measure to improve safety in reinforcement learning agents.

Findings

The new method avoids bad incentives in gridworld experiments.

Compared to simpler baselines, the proposed approach performs better in safety metrics.

Empirical results demonstrate the effectiveness of the combined baseline and deviation measure.

Abstract

How can we design safe reinforcement learning agents that avoid unnecessary disruptions to their environment? We show that current approaches to penalizing side effects can introduce bad incentives, e.g. to prevent any irreversible changes in the environment, including the actions of other agents. To isolate the source of such undesirable incentives, we break down side effects penalties into two components: a baseline state and a measure of deviation from this baseline state. We argue that some of these incentives arise from the choice of baseline, and others arise from the choice of deviation measure. We introduce a new variant of the stepwise inaction baseline and a new deviation measure based on relative reachability of states. The combination of these design choices avoids the given undesirable incentives, while simpler baselines and the unreachability measure fail. We demonstrate this empirically by comparing different combinations of baseline and deviation measure choices on a set of gridworld experiments designed to illustrate possible bad incentives.