PAC Bounds for Imitation and Model-based Batch Learning of Contextual Markov Decision Processes

TL;DR

This paper analyzes the sample complexity of imitation and model-based batch learning in contextual Markov decision processes, highlighting the advantages of imitation learning and the limitations of model-based approaches.

Contribution

It provides new sample complexity bounds for direct policy learning and demonstrates the potential impossibility and limitations of model-based learning in this setting.

Findings

Imitation learning has favorable sample complexity bounds.

Model-based learning can be impossible with finite models.

Model-based learning's complexity can grow exponentially with model class size.

Abstract

We consider the problem of batch multi-task reinforcement learning with observed context descriptors, motivated by its application to personalized medical treatment. In particular, we study two general classes of learning algorithms: direct policy learning (DPL), an imitation-learning based approach which learns from expert trajectories, and model-based learning. First, we derive sample complexity bounds for DPL, and then show that model-based learning from expert actions can, even with a finite model class, be impossible. After relaxing the conditions under which the model-based approach is expected to learn by allowing for greater coverage of state-action space, we provide sample complexity bounds for model-based learning with finite model classes, showing that there exist model classes with sample complexity exponential in their statistical complexity. We then derive a sample complexity upper bound for model-based learning based on a measure of concentration of the data distribution. Our results give formal justification for imitation learning over model-based learning in this setting.