Time-Scale Separation in Q-Learning: Extending TD($\triangle$) for Action-Value Function Decomposition

TL;DR

This paper introduces Q($ riangle$)-Learning, an extension of TD($ riangle$), which decomposes Q-functions across multiple time scales to improve stability, scalability, and convergence in long-term reinforcement learning tasks.

Contribution

It extends TD($ riangle$) to Q-Learning, enabling efficient multi-scale learning and better handling of long-term rewards in RL.

Findings

Q($ riangle$)-Learning outperforms traditional Q-Learning in benchmarks.

It achieves faster convergence on shorter time scales.

It demonstrates improved stability in deep RL environments.

Abstract

Q-Learning is a fundamental off-policy reinforcement learning (RL) algorithm that has the objective of approximating action-value functions in order to learn optimal policies. Nonetheless, it has difficulties in reconciling bias with variance, particularly in the context of long-term rewards. This paper introduces Q($\Delta$)-Learning, an extension of TD($\Delta$) for the Q-Learning framework. TD($\Delta$) facilitates efficient learning over several time scales by breaking the Q($\Delta$)-function into distinct discount factors. This approach offers improved learning stability and scalability, especially for long-term tasks where discounting bias may impede convergence. Our methodology guarantees that each element of the Q($\Delta$)-function is acquired individually, facilitating expedited convergence on shorter time scales and enhancing the learning of extended time scales. We demonstrate through theoretical analysis and practical evaluations on standard benchmarks like Atari that Q($\Delta$)-Learning surpasses conventional Q-Learning and TD learning methods in both tabular and deep RL environments.