A Lyapunov Drift-Plus-Penalty Method Tailored for Reinforcement Learning with Queue Stability

TL;DR

This paper introduces a novel Lyapunov Drift-Plus-Penalty algorithm tailored for reinforcement learning, effectively balancing queue stability and long-term optimization, with proven theoretical advantages and superior simulation performance.

Contribution

The paper develops a new LDPTRLQ algorithm that integrates Lyapunov Drift-Plus-Penalty with RL, providing theoretical guarantees and improved stability over existing methods.

Findings

LDPTRLQ outperforms baseline methods in simulations.

The algorithm effectively balances queue stability and long-term rewards.

Theoretical analysis confirms the superiority of the proposed method.

Abstract

With the proliferation of Internet of Things (IoT) devices, the demand for addressing complex optimization challenges has intensified. The Lyapunov Drift-Plus-Penalty algorithm is a widely adopted approach for ensuring queue stability, and some research has preliminarily explored its integration with reinforcement learning (RL). In this paper, we investigate the adaptation of the Lyapunov Drift-Plus-Penalty algorithm for RL applications, deriving an effective method for combining Lyapunov Drift-Plus-Penalty with RL under a set of common and reasonable conditions through rigorous theoretical analysis. Unlike existing approaches that directly merge the two frameworks, our proposed algorithm, termed Lyapunov drift-plus-penalty method tailored for reinforcement learning with queue stability (LDPTRLQ) algorithm, offers theoretical superiority by effectively balancing the greedy optimization of Lyapunov Drift-Plus-Penalty with the long-term perspective of RL. Simulation results for multiple problems demonstrate that LDPTRLQ outperforms the baseline methods using the Lyapunov drift-plus-penalty method and RL, corroborating the validity of our theoretical derivations. The results also demonstrate that our proposed algorithm outperforms other benchmarks in terms of compatibility and stability.