Safe Continual Reinforcement Learning Methods for Nonstationary Environments. Towards a Survey of the State of the Art

TL;DR

This paper surveys the current state of safe continual reinforcement learning in nonstationary environments, discussing theoretical challenges, safety mechanisms, and future directions for reliable online algorithms.

Contribution

It provides a comprehensive taxonomy and categorization of safety constraints and mechanisms in continual online safe reinforcement learning methods.

Findings

Categorizes safety constraints for online RL algorithms.

Discusses theoretical challenges and open questions.

Outlines prospects for reliable safe online learning algorithms.

Abstract

This work provides a state-of-the-art survey of continual safe online reinforcement learning (COSRL) methods. We discuss theoretical aspects, challenges, and open questions in building continual online safe reinforcement learning algorithms. We provide the taxonomy and the details of continual online safe reinforcement learning methods based on the type of safe learning mechanism that takes adaptation to nonstationarity into account. We categorize safety constraints formulation for online reinforcement learning algorithms, and finally, we discuss prospects for creating reliable, safe online learning algorithms. Keywords: safe RL in nonstationary environments, safe continual reinforcement learning under nonstationarity, HM-MDP, NSMDP, POMDP, safe POMDP, constraints for continual learning, safe continual reinforcement learning review, safe continual reinforcement learning survey, safe continual reinforcement learning, safe online learning under distribution shift, safe continual online adaptation, safe reinforcement learning, safe exploration, safe adaptation, constrained Markov decision processes, safe reinforcement learning, partially observable Markov decision process, safe reinforcement learning and hidden Markov decision processes, Safe Online Reinforcement Learning, safe online reinforcement learning, safe online reinforcement learning, safe meta-learning, safe meta-reinforcement learning, safe context-based reinforcement learning, formulating safety constraints for continual learning