Control-Oriented Model-Based Reinforcement Learning with Implicit Differentiation

TL;DR

This paper introduces an end-to-end model learning method for reinforcement learning that directly optimizes expected returns using implicit differentiation, addressing issues with likelihood-based approaches under model misspecification.

Contribution

It presents a novel implicit differentiation-based approach for model learning in reinforcement learning, improving performance when models are misspecified or limited.

Findings

Outperforms likelihood-based methods in misspecified model regimes

Provides theoretical analysis of implicit differentiation benefits

Empirical results demonstrate improved control performance

Abstract

The shortcomings of maximum likelihood estimation in the context of model-based reinforcement learning have been highlighted by an increasing number of papers. When the model class is misspecified or has a limited representational capacity, model parameters with high likelihood might not necessarily result in high performance of the agent on a downstream control task. To alleviate this problem, we propose an end-to-end approach for model learning which directly optimizes the expected returns using implicit differentiation. We treat a value function that satisfies the Bellman optimality operator induced by the model as an implicit function of model parameters and show how to differentiate the function. We provide theoretical and empirical evidence highlighting the benefits of our approach in the model misspecification regime compared to likelihood-based methods.