Offline Critic-Guided Diffusion Policy for Multi-User Delay-Constrained Scheduling

TL;DR

This paper introduces SOCD, an offline reinforcement learning algorithm with diffusion-based policy networks and critic guidance, enabling effective multi-user delay-constrained scheduling without online system interaction.

Contribution

The paper presents a novel offline RL approach with diffusion policies and critic guidance for delay-constrained scheduling, avoiding online data collection and handling complex system dynamics.

Findings

SOCD outperforms existing methods in various environments.

It is robust to system dynamics and partial observability.

Eliminates need for online system interaction during training.

Abstract

Effective multi-user delay-constrained scheduling is crucial in various real-world applications, such as instant messaging, live streaming, and data center management. In these scenarios, schedulers must make real-time decisions to satisfy both delay and resource constraints without prior knowledge of system dynamics, which are often time-varying and challenging to estimate. Current learning-based methods typically require interactions with actual systems during the training stage, which can be difficult or impractical, as it is capable of significantly degrading system performance and incurring substantial service costs. To address these challenges, we propose a novel offline reinforcement learning-based algorithm, named \underline{S}cheduling By \underline{O}ffline Learning with \underline{C}ritic Guidance and \underline{D}iffusion Generation (SOCD), to learn efficient scheduling policies purely from pre-collected \emph{offline data}. SOCD innovatively employs a diffusion-based policy network, complemented by a sampling-free critic network for policy guidance. By integrating the Lagrangian multiplier optimization into the offline reinforcement learning, SOCD effectively trains high-quality constraint-aware policies exclusively from available datasets, eliminating the need for online interactions with the system. Experimental results demonstrate that SOCD is resilient to various system dynamics, including partially observable and large-scale environments, and delivers superior performance compared to existing methods.