DynamiQ: Planning for Dynamics in Network Streaming Analytics Systems
Rohan Bhatia, Arpit Gupta, Rob Harrison, Daniel Lokshtanov, Walter, Willinger

TL;DR
DynamiQ is a new network streaming analytics platform that employs dynamic query planning to adapt to real-world traffic changes, significantly reducing resource usage compared to static planning systems.
Contribution
We introduce DynamiQ, a system with algorithms for initial query planning and runtime updates, improving resource efficiency in dynamic network workloads.
Findings
DynamiQ reduces workload by two orders of magnitude compared to Sonata.
Dynamic query planning effectively handles traffic variability.
Static planning wastes resources and limits query execution to userspace.
Abstract
The emergence of programmable data-plane targets has motivated a new hybrid design for network streaming analytics systems that combine these targets' fast packet processing speeds with the rich compute resources available at modern stream processors. However, these systems require careful query planning; that is, specifying the minute details of executing a given set of queries in a way that makes the best use of the limited resources and programmability offered by data-plane targets. We use such an existing system, Sonata, and real-world packet traces to understand how executing a fixed query workload is affected by the unknown dynamics of the traffic that defines the target's input workload. We observe that static query planning, as employed by Sonata, cannot handle even small changes in the input workload, wasting data-plane resources to the point where query execution is confined…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsCloud Computing and Resource Management · Software System Performance and Reliability · Network Security and Intrusion Detection
