Continual Reinforcement Learning for Cyber-Physical Systems: Lessons Learned and Open Challenges

TL;DR

This paper examines the challenges of applying continual reinforcement learning to cyber-physical systems like autonomous driving, highlighting issues such as catastrophic forgetting and hyperparameter sensitivity, and suggests future research directions.

Contribution

It provides empirical insights into the open challenges of continual RL in autonomous driving and proposes research questions to improve robustness and interdisciplinary approaches.

Findings

Identified key challenges: environment abstraction, hyperparameter sensitivity, catastrophic forgetting.

Demonstrated continual RL difficulties in multi-scenario autonomous parking.

Questioned neural networks' suitability for continual learning tasks.

Abstract

Continual learning (CL) is a branch of machine learning that aims to enable agents to adapt and generalise previously learned abilities so that these can be reapplied to new tasks or environments. This is particularly useful in multi-task settings or in non-stationary environments, where the dynamics can change over time. This is particularly relevant in cyber-physical systems such as autonomous driving. However, despite recent advances in CL, successfully applying it to reinforcement learning (RL) is still an open problem. This paper highlights open challenges in continual RL (CRL) based on experiments in an autonomous driving environment. In this environment, the agent must learn to successfully park in four different scenarios corresponding to parking spaces oriented at varying angles. The agent is successively trained in these four scenarios one after another, representing a CL environment, using Proximal Policy Optimisation (PPO). These experiments exposed a number of open challenges in CRL: finding suitable abstractions of the environment, oversensitivity to hyperparameters, catastrophic forgetting, and efficient use of neural network capacity. Based on these identified challenges, we present open research questions that are important to be addressed for creating robust CRL systems. In addition, the identified challenges call into question the suitability of neural networks for CL. We also identify the need for interdisciplinary research, in particular between computer science and neuroscience.