Optimal Job Scheduling and Bandwidth Augmentation in Hybrid Data Center Networks

TL;DR

This paper presents a novel optimization approach for joint job scheduling and bandwidth augmentation in hybrid data center networks, significantly reducing job completion times by up to 10% through a reformulated mixed integer nonlinear problem solved via Branch and Bound.

Contribution

It introduces a new reformulation of the complex joint scheduling and bandwidth augmentation problem enabling efficient optimal solutions with Branch and Bound.

Findings

Up to 10% reduction in job completion time.

Performance gain varies with data size.

Method outperforms existing solutions.

Abstract

Optimizing data transfers is critical for improving job performance in data-parallel frameworks. In the hybrid data center with both wired and wireless links, reconfigurable wireless links can provide additional bandwidth to speed up job execution. However, it requires the scheduler and transceivers to make joint decisions under coupled constraints. In this work, we identify that the joint job scheduling and bandwidth augmentation problem is a complex mixed integer nonlinear problem, which is not solvable by existing optimization methods. To address this bottleneck, we transform it into an equivalent problem based on the coupling of its heuristic bounds, the revised data transfer representation and non-linear constraints decoupling and reformulation, such that the optimal solution can be efficiently acquired by the Branch and Bound method. Based on the proposed method, the performance of job scheduling with and without bandwidth augmentation is studied. Experiments show that the performance gain depends on multiple factors, especially the data size. Compared with existing solutions, our method can averagely reduce the job completion time by up to 10% under the setting of production scenario.