Fast Prototyping of Distributed Stream Processing Applications with stream2gym

TL;DR

stream2gym is a tool that enables rapid prototyping and testing of distributed stream processing applications using network emulation, reducing development time and resource requirements while maintaining accuracy.

Contribution

It introduces stream2gym, a high-level interface built on Mininet for fast, resource-efficient testing of large-scale distributed stream processing applications.

Findings

stream2gym accurately reproduces results from hardware testbeds

enables testing under various network conditions

supports prototyping applications on a single laptop

Abstract

Stream processing applications have been widely adopted due to real-time data analytics demands, e.g., fraud detection, video analytics, IoT applications. Unfortunately, prototyping and testing these applications is still a cumbersome process for developers that usually requires an expensive testbed and deep multi-disciplinary expertise, including in areas such as networking, distributed systems, and data engineering. As a result, it takes a long time to deploy stream processing applications into production and yet users face several correctness and performance issues. In this paper, we present stream2gym, a tool for the fast prototyping of large-scale distributed stream processing applications. stream2gym builds on Mininet, a widely adopted network emulation platform, and provides a high-level interface to enable developers to easily test their applications under various operating conditions. We demonstrate the benefits of stream2gym by prototyping and testing several applications as well as reproducing key findings from prior research work in video analytics and network traffic monitoring. Moreover, we show stream2gym presents accurate results compared to a hardware testbed while consuming a small amount of resources (enough to be supported in a single commodity laptop even when emulating a dozen of processing nodes).