High-Accuracy Model-Based Reinforcement Learning, a Survey

TL;DR

This survey reviews recent advances in high-accuracy model-based reinforcement learning, highlighting methods that improve sample efficiency and discussing their strengths, weaknesses, and future research directions.

Contribution

It provides a comprehensive overview of diverse model-based RL methods, explaining their mechanisms and evaluating their effectiveness across different domains.

Findings

Model-based methods can achieve high accuracy with low sample complexity.

Most successful methods are domain-specific, mainly in robotics or gaming.

Future work should focus on robustness and broader applicability.

Abstract

Deep reinforcement learning has shown remarkable success in the past few years. Highly complex sequential decision making problems from game playing and robotics have been solved with deep model-free methods. Unfortunately, the sample complexity of model-free methods is often high. To reduce the number of environment samples, model-based reinforcement learning creates an explicit model of the environment dynamics. Achieving high model accuracy is a challenge in high-dimensional problems. In recent years, a diverse landscape of model-based methods has been introduced to improve model accuracy, using methods such as uncertainty modeling, model-predictive control, latent models, and end-to-end learning and planning. Some of these methods succeed in achieving high accuracy at low sample complexity, most do so either in a robotics or in a games context. In this paper, we survey these methods; we explain in detail how they work and what their strengths and weaknesses are. We conclude with a research agenda for future work to make the methods more robust and more widely applicable to other applications.