REPTILE: A Proactive Real-Time Deep Reinforcement Learning Self-adaptive Framework

TL;DR

REPTILE is a proactive, real-time deep reinforcement learning framework designed for self-adaptive software systems, predicting and responding to environmental and architectural changes before they occur.

Contribution

It introduces a novel proactive approach using deep reinforcement learning to anticipate and adapt to system novelties in real-time, including environment and architecture changes.

Findings

Predicts environmental and architectural novelties before they happen

Uses time-changing models and Markov Decision Processes for real-time adaptation

Evolves RL agent architecture based on available actions

Abstract

In this work a general framework is proposed to support the development of software systems that are able to adapt their behaviour according to the operating environment changes. The proposed approach, named REPTILE, works in a complete proactive manner and relies on Deep Reinforcement Learning-based agents to react to events, referred as novelties, that can affect the expected behaviour of the system. In our framework, two types of novelties are taken into account: those related to the context/environment and those related to the physical architecture itself. The framework, predicting those novelties before their occurrence, extracts time-changing models of the environment and uses a suitable Markov Decision Process to deal with the real-time setting. Moreover, the architecture of our RL agent evolves based on the possible actions that can be taken.