Bi-Level Online Provisioning and Scheduling with Switching Costs and Cross-Level Constraints

TL;DR

This paper introduces a novel bi-level online learning framework combining convex optimization and constrained Markov decision processes to optimize network resource provisioning and scheduling with dynamic thresholds and costs.

Contribution

It develops a new algorithm addressing bi-level online learning with switching costs and cross-level constraints, bridging the gap between OCO and CMDP models.

Findings

Achieves near-optimal regret bounds.

Ensures high-probability satisfaction of cross-level constraints.

Handles dynamic thresholds in provisioning and scheduling.

Abstract

We study a bi-level online provisioning and scheduling problem motivated by network resource allocation, where provisioning decisions are made at a slow time scale while queue-/state-dependent scheduling is performed at a fast time scale. We model this two-time-scale interaction using an upper-level online convex optimization (OCO) problem and a lower-level constrained Markov decision process (CMDP). Existing OCO typically assumes stateless decisions and thus cannot capture MDP network dynamics such as queue evolution. Meanwhile, CMDP algorithms typically assume a fixed constraint threshold, whereas in provisioning-and-scheduling systems, the threshold varies with online budget decisions. To address these gaps, we study bi-level OCO-CMDP learning under switching costs (budget reprovisioning/system reconfiguration) and cross-level constraints that couple budgets to scheduling decisions. Our new algorithm solves this learning problem via several non-trivial developments, including a carefully designed dual feedback that returns the budget multiplier as sensitivity information for the upper-level update and a lower level that solves a budget-adaptive safe exploration problem via an extended occupancy-measure linear program. We establish near-optimal regret and high-probability satisfaction of the cross-level constraints.