Model Based Meta Learning of Critics for Policy Gradients

TL;DR

This paper introduces a meta-learning framework for critics in reinforcement learning, enabling rapid adaptation to new tasks and environments by learning a critic that generalizes well across different scenarios.

Contribution

It proposes a model-based bi-level optimization algorithm to meta-learn critics that resemble ground truth Q functions, improving policy learning in new tasks without requiring a model.

Findings

Learned critics resemble ground truth Q functions.

Meta-trained critics generalize to new tasks and dynamics.

Enables model-free policy learning in unseen scenarios.

Abstract

Being able to seamlessly generalize across different tasks is fundamental for robots to act in our world. However, learning representations that generalize quickly to new scenarios is still an open research problem in reinforcement learning. In this paper we present a framework to meta-learn the critic for gradient-based policy learning. Concretely, we propose a model-based bi-level optimization algorithm that updates the critics parameters such that the policy that is learned with the updated critic gets closer to solving the meta-training tasks. We illustrate that our algorithm leads to learned critics that resemble the ground truth Q function for a given task. Finally, after meta-training, the learned critic can be used to learn new policies for new unseen task and environment settings via model-free policy gradient optimization, without requiring a model. We present results that show the generalization capabilities of our learned critic to new tasks and dynamics when used to learn a new policy in a new scenario.