Monitoring Cascading Changes of Resources in the Kubernetes Control Plane

TL;DR

This paper presents a measurement system for analyzing the propagation of cascading resource changes in the Kubernetes control plane, helping to understand and improve cluster availability.

Contribution

It introduces a novel measurement system that logs and analyzes cascading change propagation times in Kubernetes, with low resource overhead.

Findings

The system effectively measures change propagation times.

It has a small CPU and memory footprint.

Practical scenario confirms its usefulness.

Abstract

Kubernetes is a container management system that has many automated functionalities. Those functionalities are managed by configuring objects and resources in the control plane. Since most objects change their state depending on other objects' states, a change propagates to other objects in a chain. As cluster availability is influenced by the time required for these cascading changes, it is essential to make the propagations measurable and shed light on the behavior of the Kubernetes control plane. However, it is not easy because each object constantly monitors other objects' status and acts autonomously in response to their changes to play its role. In this paper, we propose a measurement system that outputs objects' change logs published from the API server in the control plane and assists in analyzing the time of cascading changes between objects by utilizing the relationships among resources. With a practical change scenario, our system is confirmed that it can measure change propagation times within a cascading change. Also, measurements on the system itself showed it has a small CPU and memory footprint.