Annual Interruption Rate as a KPI, its measurement and comparison

TL;DR

This paper explores the failure rate as a KPI, specifically the Annual Interruption Rate in Azure, and discusses methods for measuring, comparing, and statistically analyzing this metric to detect regressions and system changes.

Contribution

It introduces a comprehensive approach to measuring and comparing failure rates using statistical hypothesis testing, with practical applications in system monitoring and regression detection.

Findings

Failure rate can be effectively modeled and measured from logs.

Statistical hypothesis tests can detect regressions in failure rate.

Practical guidelines for system change analysis and validation.

Abstract

This article is divided into two chapters. The first chapter describes the failure rate as a KPI and studies its properties. The second one goes over ways to compare this KPI across two groups using the concepts of statistical hypothesis testing. In section 1., we will motivate the failure rate as a KPI (in Azure, it is dubbed `Annual Interruption Rate' or AIR. In section 3, we will discuss measuring failure rate from logs machines typically generate. In section 1.2, we will discuss the problem of measuring it from real-world data. In section 2.1, we will discuss the general concepts of hypothesis testing. In section 2.2, we will go over some general count distributions for modeling Azure reboots. In section 2.3, we will go over some experiments on applying various hypothesis tests to simulated data. In section 2.4, we will discuss some applications of this work like using these statistical methods to catch regressions in failure rate and how long we need to let changes to the system `bake' before we are reasonably sure they didn't regress failure rate.