Finite sample guarantees for quantile estimation: An application to detector threshold tuning

TL;DR

This paper provides finite sample guarantees for quantile estimators used in tuning detector thresholds for anomaly detection, ensuring controlled false alarm rates with theoretical confidence bounds validated through simulations and experiments.

Contribution

It introduces three distribution-free finite sample guarantees for quantile estimators and applies them to threshold tuning in anomaly detection.

Findings

Guarantees based on Dworetzky-Kiefer-Wolfowitz inequality

Guarantees based on Vysochanskij-Petunin inequality

Guarantees using exact beta distribution confidence intervals

Abstract

In threshold-based anomaly detection, we want to tune the threshold of a detector to achieve an acceptable false alarm rate. However, tuning the threshold is often a non-trivial task due to unknown detector output distributions. A detector threshold that provides an acceptable false alarm rate is equivalent to a specific quantile of the detector output distribution. Therefore, we use quantile estimators based on order statistics to estimate the detector threshold. The estimation of quantiles from sample data has a more than a century long tradition and we provide three different distribution-free finite sample guarantees for a class of quantile estimators. The first is based on the Dworetzky-Kiefer-Wolfowitz inequality, the second utilizes the Vysochanskij-Petunin inequality, and the third is based on exact confidence intervals for a beta distribution. These guarantees are then compared and used in the detector threshold tuning problem. We use both simulated data as well as data obtained from an experimental setup with the Temperature Control Lab to validate the guarantees provided.