Process Faster, Pay Less: Functional Isolation for Stream Processing

TL;DR

FunShare is a stream processing system that dynamically groups queries to optimize resource use, reducing costs while maintaining or improving performance through adaptive reconfiguration based on real-time metrics.

Contribution

It introduces an adaptive framework for query grouping in stream processing that balances resource efficiency with performance preservation.

Findings

Reduces resource consumption compared to isolated query execution.

Maintains or improves throughput for all queries.

Effectively adapts to changing data stream conditions.

Abstract

Concurrent workloads often extract insights from high-throughput, real-time data streams. Existing stream processing engines isolate each query's resources, ensuring robust performance but incurring high infrastructure costs. In contrast, sharing work reduces the amount of necessary resources but introduces inter-query interference, leading to performance degradation for some queries. We introduce FunShare, a stream-processing system that improves resource efficiency without compromising performance by dynamically grouping queries based on their performance characteristics. FunShare strategically relaxes query interdependencies and minimizes redundant computation while preserving individual query performance. It achieves this by using an adaptive optimization framework that monitors execution metrics, accurately estimates computation overlaps, and reconfigures execution plans on the fly in response to changes in the underlying data streams. Our evaluation demonstrates that FunShare minimizes resource consumption compared to isolated execution while maintaining or improving throughput for all queries.