Assisted Robust Reward Design

TL;DR

This paper introduces an Assisted Reward Design method that proactively exposes designers to potential failure cases during development, enabling faster and more robust reward specification for complex robotic tasks.

Contribution

It formalizes the iterative reward design process by modeling uncertainty and anticipates future evidence to improve reward robustness.

Findings

Faster improvement of robot behavior in held-out environments.

Proactively exposes designers to edge cases during development.

Enhances robustness of reward functions in autonomous driving tasks.

Abstract

Real-world robotic tasks require complex reward functions. When we define the problem the robot needs to solve, we pretend that a designer specifies this complex reward exactly, and it is set in stone from then on. In practice, however, reward design is an iterative process: the designer chooses a reward, eventually encounters an "edge-case" environment where the reward incentivizes the wrong behavior, revises the reward, and repeats. What would it mean to rethink robotics problems to formally account for this iterative nature of reward design? We propose that the robot not take the specified reward for granted, but rather have uncertainty about it, and account for the future design iterations as future evidence. We contribute an Assisted Reward Design method that speeds up the design process by anticipating and influencing this future evidence: rather than letting the designer eventually encounter failure cases and revise the reward then, the method actively exposes the designer to such environments during the development phase. We test this method in a simplified autonomous driving task and find that it more quickly improves the car's behavior in held-out environments by proposing environments that are "edge cases" for the current reward.