Double Meta-Learning for Data Efficient Policy Optimization in Non-Stationary Environments

TL;DR

This paper introduces a double meta-learning approach that combines model-based and model-free reinforcement learning techniques to efficiently learn policies in non-stationary environments, achieving high performance with less data.

Contribution

The paper proposes a novel meta-reinforcement learning method that learns a dynamic environment model for improved data efficiency and policy optimization in non-stationary settings.

Findings

Achieves data-efficient policy learning comparable to model-based methods.

Attains high asymptotic performance similar to model-free meta-reinforcement learning.

Effectively models non-stationary environments for better adaptation.

Abstract

We are interested in learning models of non-stationary environments, which can be framed as a multi-task learning problem. Model-free reinforcement learning algorithms can achieve good asymptotic performance in multi-task learning at a cost of extensive sampling, due to their approach, which requires learning from scratch. While model-based approaches are among the most data efficient learning algorithms, they still struggle with complex tasks and model uncertainties. Meta-reinforcement learning addresses the efficiency and generalization challenges on multi task learning by quickly leveraging the meta-prior policy for a new task. In this paper, we propose a meta-reinforcement learning approach to learn the dynamic model of a non-stationary environment to be used for meta-policy optimization later. Due to the sample efficiency of model-based learning methods, we are able to simultaneously train both the meta-model of the non-stationary environment and the meta-policy until dynamic model convergence. Then, the meta-learned dynamic model of the environment will generate simulated data for meta-policy optimization. Our experiment demonstrates that our proposed method can meta-learn the policy in a non-stationary environment with the data efficiency of model-based learning approaches while achieving the high asymptotic performance of model-free meta-reinforcement learning.