Launchpad: Learning to Schedule Using Offline and Online RL Methods

TL;DR

This paper introduces Launchpad, a framework that combines offline and online reinforcement learning to improve job scheduling efficiency by leveraging historical data and expert demonstrations, reducing exploration time and enhancing policy performance.

Contribution

The paper presents a novel approach that integrates offline RL and behavior cloning with online training to accelerate learning of scheduling policies from historical datasets.

Findings

Offline RL and behavior cloning can learn effective scheduling policies from logged data.

Incorporating expert demonstrations accelerates online policy learning.

The framework enables continuous improvement through online data collection.

Abstract

Deep reinforcement learning algorithms have succeeded in several challenging domains. Classic Online RL job schedulers can learn efficient scheduling strategies but often takes thousands of timesteps to explore the environment and adapt from a randomly initialized DNN policy. Existing RL schedulers overlook the importance of learning from historical data and improving upon custom heuristic policies. Offline reinforcement learning presents the prospect of policy optimization from pre-recorded datasets without online environment interaction. Following the recent success of data-driven learning, we explore two RL methods: 1) Behaviour Cloning and 2) Offline RL, which aim to learn policies from logged data without interacting with the environment. These methods address the challenges concerning the cost of data collection and safety, particularly pertinent to real-world applications of RL. Although the data-driven RL methods generate good results, we show that the performance is highly dependent on the quality of the historical datasets. Finally, we demonstrate that by effectively incorporating prior expert demonstrations to pre-train the agent, we short-circuit the random exploration phase to learn a reasonable policy with online training. We utilize Offline RL as a launchpad to learn effective scheduling policies from prior experience collected using Oracle or heuristic policies. Such a framework is effective for pre-training from historical datasets and well suited to continuous improvement with online data collection.