Black-box tests for algorithmic stability

TL;DR

This paper introduces a formal statistical framework for empirically testing the stability of algorithms in a black-box manner, providing fundamental bounds on what such tests can achieve without assumptions on data or algorithms.

Contribution

It develops a novel black-box testing framework for algorithmic stability and establishes theoretical bounds on the detection capabilities of such tests.

Findings

Framework allows stability testing without assumptions

Bounds on test effectiveness are formally derived

Applicable to complex, real-world algorithms

Abstract

Algorithmic stability is a concept from learning theory that expresses the degree to which changes to the input data (e.g., removal of a single data point) may affect the outputs of a regression algorithm. Knowing an algorithm's stability properties is often useful for many downstream applications -- for example, stability is known to lead to desirable generalization properties and predictive inference guarantees. However, many modern algorithms currently used in practice are too complex for a theoretical analysis of their stability properties, and thus we can only attempt to establish these properties through an empirical exploration of the algorithm's behavior on various data sets. In this work, we lay out a formal statistical framework for this kind of "black-box testing" without any assumptions on the algorithm or the data distribution and establish fundamental bounds on the ability of any black-box test to identify algorithmic stability.