Frisbee: automated testing of Cloud-native applications in Kubernetes

TL;DR

Frisbee is a declarative language and runtime system that automates testing of cloud-native applications on Kubernetes, enabling early detection of anomalies across application, infrastructure, and deployment levels.

Contribution

It introduces a novel declarative language and runtime for automated, comprehensive testing of cloud-native applications in Kubernetes environments.

Findings

Erroneous benchmarks can falsely indicate performance improvements.

Automated failover mechanisms need client interoperability.

Placement policies should consider clock frequency alongside cores.

Abstract

As more and more companies are migrating (or planning to migrate) from on-premise to Cloud, their focus is to find anomalies and deficits as early as possible in the development life cycle. We propose Frisbee, a declarative language and associated runtime components for testing cloud-native applications on top of Kubernetes. Given a template describing the system under test and a workflow describing the experiment, Frisbee automatically interfaces with Kubernetes to deploy the necessary software in containers, launch needed sidecars, execute the workflow steps, and perform automated checks for deviation from expected behavior. We evaluate Frisbee through a series of tests, to demonstrate its role in designing, and evaluating cloud-native applications; Frisbee helps in testing uncertainties at the level of application (e.g., dynamically changing request patterns), infrastructure (e.g., crashes, network partitions), and deployment (e.g., saturation points). Our findings have strong implications for the design, deployment, and evaluation of cloud applications. The most prominent is that: erroneous benchmark outputs can cause an apparent performance improvement, automated failover mechanisms may require interoperability with clients, and that a proper placement policy should also account for the clock frequency, not only the number of cores.