Formalizing the Problem of Side Effect Regularization

TL;DR

This paper introduces a formal framework for side effect regularization in AI, modeling the problem as a POMDP where agents balance proxy rewards with potential future tasks, validated through gridworld experiments.

Contribution

It formalizes side effect regularization using the assistance game framework and POMDPs, providing a new way to balance objectives and mitigate unintended consequences.

Findings

The POMDP model effectively captures side effect trade-offs.

Empirical validation in gridworlds supports the formalization.

Agents can balance proxy rewards with future task achievement.

Abstract

AI objectives are often hard to specify properly. Some approaches tackle this problem by regularizing the AI's side effects: Agents must weigh off "how much of a mess they make" with an imperfectly specified proxy objective. We propose a formal criterion for side effect regularization via the assistance game framework. In these games, the agent solves a partially observable Markov decision process (POMDP) representing its uncertainty about the objective function it should optimize. We consider the setting where the true objective is revealed to the agent at a later time step. We show that this POMDP is solved by trading off the proxy reward with the agent's ability to achieve a range of future tasks. We empirically demonstrate the reasonableness of our problem formalization via ground-truth evaluation in two gridworld environments.