A Study of Continual Learning Methods for Q-Learning

TL;DR

This paper empirically evaluates continual learning methods within a reinforcement learning context involving a simulated robot on a racetrack, demonstrating that CL methods can enhance learning over traditional experience replay.

Contribution

It is the first empirical comparison of CL methods in RL with non-stationary, non-disjoint subtasks, highlighting their potential benefits.

Findings

CL methods outperform experience replay in the given RL scenario

Dedicated CL methods improve learning efficiency

Non-stationary subtasks can be effectively managed with CL techniques

Abstract

We present an empirical study on the use of continual learning (CL) methods in a reinforcement learning (RL) scenario, which, to the best of our knowledge, has not been described before. CL is a very active recent research topic concerned with machine learning under non-stationary data distributions. Although this naturally applies to RL, the use of dedicated CL methods is still uncommon. This may be due to the fact that CL methods often assume a decomposition of CL problems into disjoint sub-tasks of stationary distribution, that the onset of these sub-tasks is known, and that sub-tasks are non-contradictory. In this study, we perform an empirical comparison of selected CL methods in a RL problem where a physically simulated robot must follow a racetrack by vision. In order to make CL methods applicable, we restrict the RL setting and introduce non-conflicting subtasks of known onset, which are however not disjoint and whose distribution, from the learner's point of view, is still non-stationary. Our results show that dedicated CL methods can significantly improve learning when compared to the baseline technique of "experience replay".