Providing In-network Support to Coflow Scheduling

TL;DR

This paper introduces pCoflow, a novel coflow scheduling solution that combines end-host and in-network techniques to reduce completion times in data center networks, addressing issues caused by dynamic priority changes.

Contribution

pCoflow is the first to integrate end-host coflow ordering with in-network scheduling based on packet history, improving performance over existing methods.

Findings

pCoflow reduces coflow completion time by up to 34%.

It mitigates packet re-ordering issues caused by dynamic priority changes.

The approach enhances network performance under varying load conditions.

Abstract

Many emerging distributed applications, including big data analytics, generate a number of flows that concurrently transport data across data center networks. To improve their performance, it is required to account for the behavior of a collection of flows, i.e., coflows, rather than individual. State-of-the-art solutions allow for a near-optimal completion time by continuously reordering the unfinished coflows at the end-host, using network priorities. This paper shows that dynamically changing flow priorities at the end host, without taking into account in-flight packets, can cause high-degrees of packet re-ordering, thus imposing pressure on the congestion control and potentially harming network performance in the presence of switches with shallow buffers. We present pCoflow, a new solution that integrates end-host based coflow ordering with in-network scheduling based on packet history. Our evaluation shows that pCoflow improves in CCT upon state-of-the-art solutions by up to 34% for varying load.