RetryGuard: Preventing Self-Inflicted Retry Storms in Cloud Microservices Applications

TL;DR

RetryGuard is a distributed framework designed to prevent retry storms in cloud microservices, reducing resource usage and operational costs by managing retry policies based on an analytic model.

Contribution

It introduces RetryGuard, a novel distributed control system that manages retry patterns across microservices to prevent retry storms and optimize resource utilization.

Findings

Significantly reduces resource usage and costs compared to AWS default policies.

Demonstrates scalability and performance improvements in Kubernetes with Istio.

Effectively prevents retry storms and mitigates resource contention.

Abstract

Modern cloud applications are built on independent, diverse microservices, offering scalability, flexibility, and usage-based billing. However, the structural design of these varied services, along with their reliance on auto-scalers for dynamic internet traffic, introduces significant coordination challenges. As we demonstrate in this paper, common default retry patterns used between misaligned services can turn into retry storms which drive up resource usage and costs, leading to self-inflicted Denial-of-Wallet (DoW) scenarios. To overcome these problems we introduce RetryGuard, a distributed framework for productive control of retry patterns across interdependent microservices. By managing retry policy on a per-service basis and making parallel decisions, RetryGuard prevents retry storms, curbs resource contention, and mitigates escalating operational costs. RetryGuard makes its decisions based on an analytic model that captures the relationships among retries, throughput (rejections), delays, and costs. Experimental results show that RetryGuard significantly reduces resource usage and costs compared to AWS standard and advanced retry policies. We further demonstrate its scalability and superior performance in a more complex Kubernetes deployment with the Istio service mesh, where it achieves substantial improvements.