Challenges for Using Impact Regularizers to Avoid Negative Side Effects

TL;DR

This paper critically examines the challenges and limitations of using impact regularizers in reinforcement learning to prevent negative side effects, highlighting unresolved issues and future directions.

Contribution

It provides a detailed analysis of current challenges in impact regularizers and discusses potential solutions for improving safety in reinforcement learning.

Findings

Impact regularizers can mitigate some side effects but face significant challenges.

Many design decisions influence the effectiveness of impact regularizers.

Future research directions are identified to address unresolved issues.

Abstract

Designing reward functions for reinforcement learning is difficult: besides specifying which behavior is rewarded for a task, the reward also has to discourage undesired outcomes. Misspecified reward functions can lead to unintended negative side effects, and overall unsafe behavior. To overcome this problem, recent work proposed to augment the specified reward function with an impact regularizer that discourages behavior that has a big impact on the environment. Although initial results with impact regularizers seem promising in mitigating some types of side effects, important challenges remain. In this paper, we examine the main current challenges of impact regularizers and relate them to fundamental design decisions. We discuss in detail which challenges recent approaches address and which remain unsolved. Finally, we explore promising directions to overcome the unsolved challenges in preventing negative side effects with impact regularizers.