TIMRL: A Novel Meta-Reinforcement Learning Framework for Non-Stationary and Multi-Task Environments

TL;DR

This paper introduces TIMRL, a meta-reinforcement learning framework that uses Gaussian mixture models and transformers to better adapt to non-stationary and multi-task environments, improving task inference and sample efficiency.

Contribution

The paper proposes a novel meta-RL method combining Gaussian mixture models and transformers for explicit task encoding in non-stationary environments.

Findings

Significantly improves sample efficiency in non-stationary environments

Accurately classifies and recognizes multiple tasks

Outperforms existing methods on MuJoCo benchmarks

Abstract

In recent years, meta-reinforcement learning (meta-RL) algorithm has been proposed to improve sample efficiency in the field of decision-making and control, enabling agents to learn new knowledge from a small number of samples. However, most research uses the Gaussian distribution to extract task representation, which is poorly adapted to tasks that change in non-stationary environment. To address this problem, we propose a novel meta-reinforcement learning method by leveraging Gaussian mixture model and the transformer network to construct task inference model. The Gaussian mixture model is utilized to extend the task representation and conduct explicit encoding of tasks. Specifically, the classification of tasks is encoded through transformer network to determine the Gaussian component corresponding to the task. By leveraging task labels, the transformer network is trained using supervised learning. We validate our method on MuJoCo benchmarks with non-stationary and multi-task environments. Experimental results demonstrate that the proposed method dramatically improves sample efficiency and accurately recognizes the classification of the tasks, while performing excellently in the environment.