Model-Advantage and Value-Aware Models for Model-Based Reinforcement Learning: Bridging the Gap in Theory and Practice

TL;DR

This paper demonstrates that value-aware model learning is both theoretically advantageous and practically effective for continuous control tasks in model-based reinforcement learning, bridging the gap between theory and practice.

Contribution

It introduces a novel value-aware model learning objective and addresses the issue of stale value estimates, enabling successful deployment in real-world continuous control tasks.

Findings

Value-aware models outperform traditional models in continuous control environments.

The proposed method works with minimal modifications to existing algorithms.

Better performance than baseline models in several robotic tasks.

Abstract

This work shows that value-aware model learning, known for its numerous theoretical benefits, is also practically viable for solving challenging continuous control tasks in prevalent model-based reinforcement learning algorithms. First, we derive a novel value-aware model learning objective by bounding the model-advantage i.e. model performance difference, between two MDPs or models given a fixed policy, achieving superior performance to prior value-aware objectives in most continuous control environments. Second, we identify the issue of stale value estimates in naively substituting value-aware objectives in place of maximum-likelihood in dyna-style model-based RL algorithms. Our proposed remedy to this issue bridges the long-standing gap in theory and practice of value-aware model learning by enabling successful deployment of all value-aware objectives in solving several continuous control robotic manipulation and locomotion tasks. Our results are obtained with minimal modifications to two popular and open-source model-based RL algorithms -- SLBO and MBPO, without tuning any existing hyper-parameters, while also demonstrating better performance of value-aware objectives than these baseline in some environments.