Model Embedding Model-Based Reinforcement Learning

TL;DR

This paper introduces MEMB, a model-embedding model-based reinforcement learning algorithm that balances data efficiency and model bias by integrating real and imaginary data, achieving state-of-the-art results.

Contribution

The paper proposes a novel MEMB algorithm that embeds the model into policy updates within a probabilistic framework, balancing data efficiency and bias.

Findings

MEMB achieves state-of-the-art performance on benchmarks.

Theoretical analysis supports MEMB's effectiveness under Lipschitz assumptions.

Embedding models in policy updates improves reinforcement learning efficiency.

Abstract

Model-based reinforcement learning (MBRL) has shown its advantages in sample-efficiency over model-free reinforcement learning (MFRL). Despite the impressive results it achieves, it still faces a trade-off between the ease of data generation and model bias. In this paper, we propose a simple and elegant model-embedding model-based reinforcement learning (MEMB) algorithm in the framework of the probabilistic reinforcement learning. To balance the sample-efficiency and model bias, we exploit both real and imaginary data in the training. In particular, we embed the model in the policy update and learn $Q$ and $V$ functions from the real data set. We provide the theoretical analysis of MEMB with the Lipschitz continuity assumption on the model and policy. At last, we evaluate MEMB on several benchmarks and demonstrate our algorithm can achieve state-of-the-art performance.