On-Robot Bayesian Reinforcement Learning for POMDPs

TL;DR

This paper introduces a Bayesian reinforcement learning framework tailored for robotics, leveraging factored representations and Monte-Carlo methods to efficiently learn in uncertain, real-world environments with minimal data.

Contribution

It develops a specialized Bayesian RL approach for physical systems, incorporating factored models and a sample-based solution for on-robot learning.

Findings

Achieved near-optimal performance after few real-world episodes.

Effectively handled uncertainty in human-robot interaction tasks.

Demonstrated efficiency with on-robot learning in complex environments.

Abstract

Robot learning is often difficult due to the expense of gathering data. The need for large amounts of data can, and should, be tackled with effective algorithms and leveraging expert information on robot dynamics. Bayesian reinforcement learning (BRL), thanks to its sample efficiency and ability to exploit prior knowledge, is uniquely positioned as such a solution method. Unfortunately, the application of BRL has been limited due to the difficulties of representing expert knowledge as well as solving the subsequent inference problem. This paper advances BRL for robotics by proposing a specialized framework for physical systems. In particular, we capture this knowledge in a factored representation, then demonstrate the posterior factorizes in a similar shape, and ultimately formalize the model in a Bayesian framework. We then introduce a sample-based online solution method, based on Monte-Carlo tree search and particle filtering, specialized to solve the resulting model. This approach can, for example, utilize typical low-level robot simulators and handle uncertainty over unknown dynamics of the environment. We empirically demonstrate its efficiency by performing on-robot learning in two human-robot interaction tasks with uncertainty about human behavior, achieving near-optimal performance after only a handful of real-world episodes. A video of learned policies is at https://youtu.be/H9xp60ngOes.