Decouple and Decompose: Scaling Resource Allocation with DeDe

TL;DR

DeDe is a scalable optimization framework that leverages problem structure to efficiently solve large-scale resource allocation problems in cloud systems, outperforming existing methods in speed and quality.

Contribution

The paper introduces DeDe, a novel decouple-and-decompose approach that exploits problem separability to enable parallel, efficient resource allocation solutions at large scale.

Findings

DeDe achieves significant speedups in resource allocation tasks.

DeDe produces higher-quality allocations compared to baseline methods.

DeDe is applicable to diverse cloud resource management problems.

Abstract

Efficient resource allocation is essential in cloud systems to facilitate resource sharing among tenants. However, the growing scale of these optimization problems have outpaced commercial solvers commonly employed in production. To accelerate resource allocation, prior approaches either customize solutions for narrow domains or impose workload-specific assumptions. In this work, we revisit real-world resource allocation problems and uncover a common underlying structure: the vast majority of these problems are inherently separable, i.e., they optimize the aggregate utility of individual resource and demand allocations, under separate constraints for each resource and each demand. Building on this observation, we develop DeDe, a scalable and theoretically rooted optimization framework for large-scale resource allocation. At the core of DeDe is a decouple-and-decompose approach: it decouples entangled resource and demand constraints and thereby decomposes the overall optimization into alternating per-resource and per-demand subproblems that can be solved efficiently and in parallel. We have implemented and released DeDe as a Python package with a familiar modeling interface. Our experiments on three representative resource allocation tasks -- cluster scheduling, traffic engineering, and load balancing -- demonstrate that DeDe delivers significant speedups while generating higher-quality allocations.