# Metaflow: A DAG-Based Network Abstraction for Distributed Applications

## TL;DR

This paper introduces Metaflow, a new DAG-based network abstraction that better captures communication dependencies in distributed applications, leading to improved scheduling performance over existing flow and coflow models.

## Contribution

The paper proposes Metaflow, an expressive DAG-based abstraction that bridges the gap between flow and coflow models for better network scheduling in distributed systems.

## Key findings

- Metaflow-based scheduling outperforms coflow-based algorithms by 1.78x.
- Metaflow captures communication dependencies more effectively.
- Evaluation demonstrates improved application performance.

## Abstract

In the past decade, increasingly network scheduling techniques have been proposed to boost the distributed application performance. Flow-level metrics, such as flow completion time (FCT), are based on the abstraction of flows yet they cannot capture the semantics of communication in a cluster application. Being aware of this problem, coflow is proposed as a new network abstraction. However, it is insufficient to reveal the dependencies between computation and communication. As a result, the real application performance can be hurt, especially in the absence of hard barriers. Based on the computation DAG of the application, we propose an expressive abstraction namely metaflow that resides in the middle of the two extreme points of flows and coflows. Evaluation results show that metaflow-based scheduling can outperform the coflow-based algorithm by 1.78x.