A Contraction Approach to Model-based Reinforcement Learning

TL;DR

This paper introduces a contraction-based theoretical framework for model-based reinforcement learning, analyzing reward errors in continuous spaces and highlighting the benefits of branched rollouts and GAN-type imitation learning.

Contribution

It provides a novel contraction approach to analyze reward errors, applicable to both stochastic and deterministic transitions, and compares GAN-based imitation to behavioral cloning.

Findings

Branched rollouts reduce reward error in deterministic transitions.

The contraction approach recovers quadratic error bounds.

GAN-type imitation learning outperforms behavioral cloning with a well-trained discriminator.

Abstract

Despite its experimental success, Model-based Reinforcement Learning still lacks a complete theoretical understanding. To this end, we analyze the error in the cumulative reward using a contraction approach. We consider both stochastic and deterministic state transitions for continuous (non-discrete) state and action spaces. This approach doesn't require strong assumptions and can recover the typical quadratic error to the horizon. We prove that branched rollouts can reduce this error and are essential for deterministic transitions to have a Bellman contraction. Our analysis of policy mismatch error also applies to Imitation Learning. In this case, we show that GAN-type learning has an advantage over Behavioral Cloning when its discriminator is well-trained.