Leveraging Demonstrations to Improve Online Learning: Quality Matters

TL;DR

This paper explores how offline demonstration data, especially of varying quality, can enhance online learning performance using Thompson sampling in multi-armed bandits, with theoretical and empirical insights.

Contribution

It introduces an informed Thompson sampling algorithm that incorporates demonstration data via Bayesian methods and provides regret bounds dependent on demonstration quality.

Findings

Higher demonstration quality leads to greater online performance improvements.

The proposed Bayesian bootstrapping method effectively reduces empirical regret.

Pretraining with expert demonstrations significantly benefits online learning outcomes.

Abstract

We investigate the extent to which offline demonstration data can improve online learning. It is natural to expect some improvement, but the question is how, and by how much? We show that the degree of improvement must depend on the quality of the demonstration data. To generate portable insights, we focus on Thompson sampling (TS) applied to a multi-armed bandit as a prototypical online learning algorithm and model. The demonstration data is generated by an expert with a given competence level, a notion we introduce. We propose an informed TS algorithm that utilizes the demonstration data in a coherent way through Bayes' rule and derive a prior-dependent Bayesian regret bound. This offers insight into how pretraining can greatly improve online performance and how the degree of improvement increases with the expert's competence level. We also develop a practical, approximate informed TS algorithm through Bayesian bootstrapping and show substantial empirical regret reduction through experiments.