High-level Stream Processing: A Complementary Analysis of Fault Recovery

TL;DR

This paper analyzes fault recovery in high-level stream processing frameworks, highlighting the potential for improvements and the need for better configuration management and abstractions for large-scale, real-time analytics systems.

Contribution

It extends fault recovery measurements with new experiments and explores configuration space, emphasizing the need for transparent tuning abstractions in industry-scale deployments.

Findings

Significant potential for fault recovery improvements

Configuration complexity challenges in tuning

Need for new abstractions for scalable deployment

Abstract

Parallel computing is very important to accelerate the performance of software systems. Additionally, considering that a recurring challenge is to process high data volumes continuously, stream processing emerged as a paradigm and software architectural style. Several software systems rely on stream processing to deliver scalable performance, whereas open-source frameworks provide coding abstraction and high-level parallel computing. Although stream processing's performance is being extensively studied, the measurement of fault tolerance--a key abstraction offered by stream processing frameworks--has still not been adequately measured with comprehensive testbeds. In this work, we extend the previous fault recovery measurements with an exploratory analysis of the configuration space, additional experimental measurements, and analysis of improvement opportunities. We focus on robust deployment setups inspired by requirements for near real-time analytics of a large cloud observability platform. The results indicate significant potential for improving fault recovery and performance. However, these improvements entail grappling with configuration complexities, particularly in identifying and selecting the configurations to be fine-tuned and determining the appropriate values for them. Therefore, new abstractions for transparent configuration tuning are also needed for large-scale industry setups. We believe that more software engineering efforts are needed to provide insights into potential abstractions and how to achieve them. The stream processing community and industry practitioners could also benefit from more interactions with the high-level parallel programming community, whose expertise and insights on making parallel programming more productive and efficient could be extended.