Banyan: A Scoped Dataflow Engine for Graph Query Service

TL;DR

Banyan introduces a scoped dataflow engine for graph query services that enhances control, scheduling, and scalability, significantly outperforming existing engines in large-scale graph data processing.

Contribution

The paper presents Banyan, a novel scoped dataflow model and engine for graph query services, addressing control and scheduling challenges with substantial performance improvements.

Findings

Up to 1000x performance improvement over existing engines

Effective performance isolation and load balancing achieved

Scalable on both single machine and distributed environments

Abstract

Graph query services (GQS) are widely used today to interactively answer graph traversal queries on large-scale graph data. Existing graph query engines focus largely on optimizing the latency of a single query. This ignores significant challenges posed by GQS, including fine-grained control and scheduling during query execution, as well as performance isolation and load balancing in various levels from across user to intra-query. To tackle these control and scheduling challenges, we propose a novel scoped dataflow for modeling graph traversal queries, which explicitly exposes concurrent execution and control of any subquery to the finest granularity. We implemented Banyan, an engine based on the scoped dataflow model for GQS. Banyan focuses on scaling up the performance on a single machine, and provides the ability to easily scale out. Extensive experiments on multiple benchmarks show that Banyan improves performance by up to three orders of magnitude over state-of-the-art graph query engines, while providing performance isolation and load balancing.