StreamShield: A Production-Proven Resiliency Solution for Apache Flink at ByteDance

TL;DR

StreamShield is a comprehensive resiliency solution for Apache Flink at ByteDance, combining runtime optimization, fault-tolerance, hybrid replication, and high availability to ensure stability and rapid recovery in large-scale production environments.

Contribution

It introduces a production-proven, integrated resiliency framework for Apache Flink, addressing operational challenges and enhancing fault tolerance in large-scale deployments.

Findings

Improved fault recovery times in production clusters

Enhanced system stability under failure conditions

Effective deployment pipeline ensuring reliability

Abstract

Distributed Stream Processing Systems (DSPSs) form the backbone of real-time processing and analytics at ByteDance, where Apache Flink powers one of the largest production clusters worldwide. Ensuring resiliency, the ability to withstand and rapidly recover from failures, together with operational stability, which provides consistent and predictable performance under normal conditions, is essential for meeting strict Service Level Objectives (SLOs). However, achieving resiliency and stability in large-scale production environments remains challenging due to the cluster scale, business diversity, and significant operational overhead. In this work, we present StreamShield, a production-proven resiliency solution deployed in ByteDance's Flink clusters. Designed along complementary perspectives of the engine and cluster, StreamShield introduces key techniques to enhance resiliency, covering runtime optimization, fine-grained fault-tolerance, hybrid replication strategy, and high availability under external systems. Furthermore, StreamShield proposes a robust testing and deployment pipeline that ensures reliability and robustness in production releases. Extensive evaluations on a production cluster demonstrate the efficiency and effectiveness of techniques proposed by StreamShield.