Reward Reports for Reinforcement Learning

TL;DR

This paper introduces Reward Reports, a dynamic documentation framework for reinforcement learning systems that tracks ongoing updates, feedback effects, and societal impacts post-deployment, inspired by ML documentation practices.

Contribution

It proposes a novel living document framework for documenting and tracking the evolving design, assumptions, and societal effects of reinforcement learning systems after deployment.

Findings

Reward Reports effectively track system updates and feedback effects.

Application to Meta's BlenderBot 3 demonstrates practical utility.

Includes examples from various domains like gaming, recommendation, and traffic control.

Abstract

Building systems that are good for society in the face of complex societal effects requires a dynamic approach. Recent approaches to machine learning (ML) documentation have demonstrated the promise of discursive frameworks for deliberation about these complexities. However, these developments have been grounded in a static ML paradigm, leaving the role of feedback and post-deployment performance unexamined. Meanwhile, recent work in reinforcement learning has shown that the effects of feedback and optimization objectives on system behavior can be wide-ranging and unpredictable. In this paper we sketch a framework for documenting deployed and iteratively updated learning systems, which we call Reward Reports. Taking inspiration from various contributions to the technical literature on reinforcement learning, we outline Reward Reports as living documents that track updates to design choices and assumptions behind what a particular automated system is optimizing for. They are intended to track dynamic phenomena arising from system deployment, rather than merely static properties of models or data. After presenting the elements of a Reward Report, we discuss a concrete example: Meta's BlenderBot 3 chatbot. Several others for game-playing (DeepMind's MuZero), content recommendation (MovieLens), and traffic control (Project Flow) are included in the appendix.