# Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning

## TL;DR

This paper introduces model-based value expansion, a method that leverages learned dynamics models with controlled uncertainty to improve value estimation and reduce sample complexity in reinforcement learning.

## Contribution

It presents a novel approach that allows wider use of learned dynamics models in model-free RL by controlling for uncertainty through fixed-depth imagination.

## Key findings

- Improved value estimation in continuous control tasks.
- Reduced sample complexity in reinforcement learning.
- Enhanced utilization of learned dynamics models.

## Abstract

Recent model-free reinforcement learning algorithms have proposed incorporating learned dynamics models as a source of additional data with the intention of reducing sample complexity. Such methods hold the promise of incorporating imagined data coupled with a notion of model uncertainty to accelerate the learning of continuous control tasks. Unfortunately, they rely on heuristics that limit usage of the dynamics model. We present model-based value expansion, which controls for uncertainty in the model by only allowing imagination to fixed depth. By enabling wider use of learned dynamics models within a model-free reinforcement learning algorithm, we improve value estimation, which, in turn, reduces the sample complexity of learning.