The EnvDesign Model: A Method to Solve the Environment Design Problem

TL;DR

The paper introduces the EnvDesign model, a novel method using graph theory and optimization to design diverse, priority-aware testing environments for cloud platform reliability, applicable to combinatorial testing in various domains.

Contribution

It presents the EnvDesign model, a new approach that formulates environment design as an optimization problem using graph theory, supporting diverse and prioritized testing environments.

Findings

EnvDesign effectively models environment diversity and priorities.

The method improves testing coverage and reliability.

Supports combinatorial testing across domains.

Abstract

Today, several people and organizations rely on cloud platforms. The reliability of cloud platforms depends heavily on the performance of their internal programs (agents). To better prevent regressions in cloud platforms, the design of pre-production testing environments (that test new agents, new hardwares, and other changes) must take into account the diversity of server/node properties (hardware model, virtual machine type, etc.) across the fleet and dynamically emphasize or de-emphasize the prevalence of certain node properties based on current testing priorities. This paper formulates this task as the ``environment design" problem and presents the EnvDesign model, a method that uses graph theory and optimization algorithms to solve the environment design problem. The EnvDesign model was built on context and techniques that apply to combinatorial testing in general, so it can support combinatorial testing in other domains. An earlier version of this paper was peer-reviewed and published internally at Microsoft.