Optimizing Co-flows Scheduling and Routing in Data Centre Networks for Big Data Applications

TL;DR

This paper presents an optimization framework for co-flow scheduling and routing in data center networks, focusing on energy efficiency and completion time, using MILP models for various architectures including PON-based designs.

Contribution

It introduces a MILP-based optimization approach for co-flow scheduling in both traditional and PON-based data center architectures, considering energy and time objectives.

Findings

Optimized scheduling reduces energy consumption and completion time.

PON-based architectures outperform traditional electronic switching in efficiency.

Traffic patterns significantly impact scheduling effectiveness.

Abstract

This paper optimizes the scheduling and routing of the co-flows of MapReduce shuffling phase in state-of-the-art and proposed Passive Optical Networking (PON)-based Data Centre Network (DCN) architectures. A time-slotted Mixed Integer Linear Programming (MILP) model is developed and used for the optimization with the objective of minimizing either the total energy consumption or the completion time. The DCN architectures include four state-of-the-art electronic switching architectures which are spine-leaf, Fat-tree, BCube, and DCell data centres. The proposed PON-based DCN architectures include two designs that utilize ports in Optical Line Terminal (OLT) line cards for inter and possibly intra data centre networking in addition to passive interconnects for the intra data centre networking between different PON groups (i.e. racks) within a PON cell (i.e. number of PON groups connected to a single OLT port). The first design is a switch-centric design that uses two Arrayed Waveguide Grating Routers (AWGRs) and the second is a server-centric design. The study also considers different traffic patterns defined according to the distribution of map and reduce tasks in the servers and data skewness.