Dynamic Horizon Value Estimation for Model-based Reinforcement Learning

TL;DR

This paper introduces DMVE, an adaptive method for model-based reinforcement learning that dynamically adjusts the rollout horizon using a reconstruction-based world model, leading to improved sample efficiency and performance.

Contribution

We propose DMVE, a novel adaptive horizon value expansion method that uses reconstruction-based world models to improve value estimation in visual control tasks.

Findings

DMVE outperforms baselines in sample efficiency.

DMVE achieves higher final performance.

Adaptive horizon adjustment improves value estimation accuracy.

Abstract

Existing model-based value expansion methods typically leverage a world model for value estimation with a fixed rollout horizon to assist policy learning. However, the fixed rollout with an inaccurate model has a potential to harm the learning process. In this paper, we investigate the idea of using the model knowledge for value expansion adaptively. We propose a novel method called Dynamic-horizon Model-based Value Expansion (DMVE) to adjust the world model usage with different rollout horizons. Inspired by reconstruction-based techniques that can be applied for visual data novelty detection, we utilize a world model with a reconstruction module for image feature extraction, in order to acquire more precise value estimation. The raw and the reconstructed images are both used to determine the appropriate horizon for adaptive value expansion. On several benchmark visual control tasks, experimental results show that DMVE outperforms all baselines in sample efficiency and final performance, indicating that DMVE can achieve more effective and accurate value estimation than state-of-the-art model-based methods.