Learning from Outside the Viability Kernel: Why we Should Build Robots that can Fall with Grace

TL;DR

This paper argues that initializing robots in unviable states can improve learning by shaping the reward landscape, challenging traditional approaches that avoid such states.

Contribution

It demonstrates that including unviable initializations can enhance reinforcement learning efficiency in robotics by influencing the reward landscape.

Findings

Unviable initializations can positively impact learning.

Reward landscape is significantly affected by initial state choices.

Including failing states can improve policy robustness.

Abstract

Despite impressive results using reinforcement learning to solve complex problems from scratch, in robotics this has still been largely limited to model-based learning with very informative reward functions. One of the major challenges is that the reward landscape often has large patches with no gradient, making it difficult to sample gradients effectively. We show here that the robot state-initialization can have a more important effect on the reward landscape than is generally expected. In particular, we show the counter-intuitive benefit of including initializations that are unviable, in other words initializing in states that are doomed to fail.