Towards Adapting Reinforcement Learning Agents to New Tasks: Insights from Q-Values

TL;DR

This paper investigates how the information retained in Q-values by DQNs can be leveraged to adapt reinforcement learning agents to new tasks more efficiently, with experiments on environments and autonomous vehicle scenarios.

Contribution

It offers new insights into the relationship between Q-value accuracy and sample-efficient adaptation in reinforcement learning, highlighting the importance of training algorithms.

Findings

Closer Q-value estimates lead to faster adaptation.

Training algorithms significantly affect Q-value accuracy.

Sample efficiency improves with better initial Q-values.

Abstract

While contemporary reinforcement learning research and applications have embraced policy gradient methods as the panacea of solving learning problems, value-based methods can still be useful in many domains as long as we can wrangle with how to exploit them in a sample efficient way. In this paper, we explore the chaotic nature of DQNs in reinforcement learning, while understanding how the information that they retain when trained can be repurposed for adapting a model to different tasks. We start by designing a simple experiment in which we are able to observe the Q-values for each state and action in an environment. Then we train in eight different ways to explore how these training algorithms affect the way that accurate Q-values are learned (or not learned). We tested the adaptability of each trained model when retrained to accomplish a slightly modified task. We then scaled our setup to test the larger problem of an autonomous vehicle at an unprotected intersection. We observed that the model is able to adapt to new tasks quicker when the base model's Q-value estimates are closer to the true Q-values. The results provide some insights and guidelines into what algorithms are useful for sample efficient task adaptation.