MetaFlow: a Scalable Metadata Lookup Service for Distributed File Systems in Data Centers

TL;DR

MetaFlow introduces a scalable, SDN-based metadata lookup service for distributed file systems, significantly improving throughput and reducing latency by efficiently distributing lookup workloads over network switches.

Contribution

The paper presents MetaFlow, a novel SDN-enabled metadata lookup system that overcomes bottlenecks in DHT-based systems by leveraging physical topology and flow tables.

Findings

Increases system throughput by up to 3.2 times.

Reduces system latency by up to 5 times.

Demonstrates significant performance gains in both simulation and real deployment.

Abstract

In large-scale distributed file systems, efficient meta- data operations are critical since most file operations have to interact with metadata servers first. In existing distributed hash table (DHT) based metadata management systems, the lookup service could be a performance bottleneck due to its significant CPU overhead. Our investigations showed that the lookup service could reduce system throughput by up to 70%, and increase system latency by a factor of up to 8 compared to ideal scenarios. In this paper, we present MetaFlow, a scalable metadata lookup service utilizing software-defined networking (SDN) techniques to distribute lookup workload over network components. MetaFlow tackles the lookup bottleneck problem by leveraging B-tree, which is constructed over the physical topology, to manage flow tables for SDN-enabled switches. Therefore, metadata requests can be forwarded to appropriate servers using only switches. Extensive performance evaluations in both simulations and testbed showed that MetaFlow increases system throughput by a factor of up to 3.2, and reduce system latency by a factor of up to 5 compared to DHT-based systems. We also deployed MetaFlow in a distributed file system, and demonstrated significant performance improvement.