Robust Restless Multi-Armed Bandit for Data Center Flexibility Services Through Virtual Machine Scheduling

TL;DR

This paper introduces a robust RMAB framework for data center VM scheduling to enhance grid demand response, utilizing a mixed strategy with Whittle-index policies and UCB to improve learning efficiency and robustness.

Contribution

It develops a novel mixed-strategy RMAB approach combining Whittle-index policies with UCB to address uncertainties in VM rescheduling for demand response.

Findings

The mixed-strategy algorithm outperforms pure Thompson-Whittle in noisy conditions.

The approach remains robust across different state-space sizes.

It surpasses the performance of the EXP4 framework.

Abstract

Energy demands from data centers have surged and stressed the grid in recent years. Electric grids require balancing supply and demand every second, motivating demand response (reduction) from large loads, including data centers. This can be achieved by rescheduling jobs on physical machines. Its real-time implementation is uncertain due to fluctuating resource utilization, and rescheduling incurs quality-of-service (QoS) losses that providers are unwilling to disclose. We propose a restless multi-arm bandit (RMAB) framework in which the grid operator requests load reductions without access to detailed job-rescheduling procedures. Using the open-source virtual machine (VM) datasets, we model job arrivals and rescheduling at each data center as a restless arm in a Markov decision process (MDP), and derive Whittle-index-based policies based on the learned transition function via Thompson sampling. To overcome the weakness of an increasingly long learning process due to an enlarged state space, we used a mixed strategy that included a global upper confidence bound (UCB) encoded with trust indices to enhance robustness and accelerate learning. Results show that the proposed mixed-strategy algorithm remains robust across varying state-space sizes and consistently outperforms the pure Thompson-Whittle (TW) algorithm, especially when contextual information is noisy. It also demonstrates superior performance compared to the state-of-the-art EXP4 framework. We provided an open-sourced code for reproducibility.