Adapting the Behavior of Reinforcement Learning Agents to Changing Action Spaces and Reward Functions

TL;DR

This paper presents MORPHIN, a self-adaptive Q-learning framework that dynamically adjusts to changing reward functions and action spaces in non-stationary environments, enhancing learning efficiency and stability.

Contribution

MORPHIN introduces a novel method for real-time adaptation of RL agents to environmental changes without full retraining, combining concept drift detection with hyperparameter adjustments.

Findings

MORPHIN outperforms standard Q-learning in convergence speed.

It maintains performance despite changes in reward functions and action spaces.

Learning efficiency improves by up to 1.7 times.

Abstract

Reinforcement Learning (RL) agents often struggle in real-world applications where environmental conditions are non-stationary, particularly when reward functions shift or the available action space expands. This paper introduces MORPHIN, a self-adaptive Q-learning framework that enables on-the-fly adaptation without full retraining. By integrating concept drift detection with dynamic adjustments to learning and exploration hyperparameters, MORPHIN adapts agents to changes in both the reward function and on-the-fly expansions of the agent's action space, while preserving prior policy knowledge to prevent catastrophic forgetting. We validate our approach using a Gridworld benchmark and a traffic signal control simulation. The results demonstrate that MORPHIN achieves superior convergence speed and continuous adaptation compared to a standard Q-learning baseline, improving learning efficiency by up to 1.7x.