A Meta Reinforcement Learning-based Approach for Self-Adaptive System

TL;DR

This paper introduces a meta-reinforcement learning approach for self-adaptive systems that efficiently learns and adapts to multiple environment models, demonstrated through a robotic case study.

Contribution

It presents a novel meta-reinforcement learning method for online adaptation in self-learning systems, addressing incomplete information and multiple environment models.

Findings

Meta policy quickly adapts to real environment dynamics.

Improved efficiency in model construction for adaptation.

Successful application in a robotic system case study.

Abstract

A self-learning adaptive system (SLAS) uses machine learning to enable and enhance its adaptability. Such systems are expected to perform well in dynamic situations. For learning high-performance adaptation policy, some assumptions must be made on the environment-system dynamics when information about the real situation is incomplete. However, these assumptions cannot be expected to be always correct, and yet it is difficult to enumerate all possible assumptions. This leads to the problem of incomplete-information learning. We consider this problem as multiple model problem in terms of finding the adaptation policy that can cope with multiple models of environment-system dynamics. This paper proposes a novel approach to engineering the online adaptation of SLAS. It separates three concerns that are related to the adaptation policy and presents the modeling and synthesis process, with the goal of achieving higher model construction efficiency. In addition, it designs a meta-reinforcement learning algorithm for learning the meta policy over the multiple models, so that the meta policy can quickly adapt to the real environment-system dynamics. At last, it reports the case study on a robotic system to evaluate the adaptability of the approach.