DynamiQ: Planning for Dynamics in Network Streaming Analytics Systems

TL;DR

DynamiQ is a new network streaming analytics platform that employs dynamic query planning to adapt to real-world traffic changes, significantly reducing resource usage compared to static planning systems.

Contribution

We introduce DynamiQ, a system with algorithms for initial query planning and runtime updates, improving resource efficiency in dynamic network workloads.

Findings

DynamiQ reduces workload by two orders of magnitude compared to Sonata.

Dynamic query planning effectively handles traffic variability.

Static planning wastes resources and limits query execution to userspace.

Abstract

The emergence of programmable data-plane targets has motivated a new hybrid design for network streaming analytics systems that combine these targets' fast packet processing speeds with the rich compute resources available at modern stream processors. However, these systems require careful query planning; that is, specifying the minute details of executing a given set of queries in a way that makes the best use of the limited resources and programmability offered by data-plane targets. We use such an existing system, Sonata, and real-world packet traces to understand how executing a fixed query workload is affected by the unknown dynamics of the traffic that defines the target's input workload. We observe that static query planning, as employed by Sonata, cannot handle even small changes in the input workload, wasting data-plane resources to the point where query execution is confined mainly to userspace. This paper presents the design and implementation of DynamiQ, a new network streaming analytics platform that employs dynamic query planning to deal with the dynamics of real-world input workloads. Specifically, we develop a suite of practical algorithms for (i) computing effective initial query plans (to start query execution) and (ii) enabling efficient updating of portions of such an initial query plan at runtime (to adapt to changes in the input workload). Using real-world packet traces as input workload, we show that compared to Sonata, DynamiQ reduces the stream processor's workload by two orders of magnitude.