Is Algorithmic Stability Testable? A Unified Framework under Computational Constraints

TL;DR

This paper investigates the fundamental limits of testing algorithmic stability in machine learning, showing that under computational constraints, exhaustive search is the only viable method, making stability testing generally infeasible.

Contribution

The authors introduce a unified framework that characterizes the hardness of testing algorithmic stability across diverse data spaces, highlighting the impracticality of stability verification under limited data and computational resources.

Findings

Testing stability is impossible with limited data in uncountably infinite spaces.

Exhaustive search is the only universal method for certifying stability under constraints.

Fundamental limits exist on the ability to test stability for black-box algorithms.

Abstract

Algorithmic stability is a central notion in learning theory that quantifies the sensitivity of an algorithm to small changes in the training data. If a learning algorithm satisfies certain stability properties, this leads to many important downstream implications, such as generalization, robustness, and reliable predictive inference. Verifying that stability holds for a particular algorithm is therefore an important and practical question. However, recent results establish that testing the stability of a black-box algorithm is impossible, given limited data from an unknown distribution, in settings where the data lies in an uncountably infinite space (such as real-valued data). In this work, we extend this question to examine a far broader range of settings, where the data may lie in any space -- for example, categorical data. We develop a unified framework for quantifying the hardness of testing algorithmic stability, which establishes that across all settings, if the available data is limited then exhaustive search is essentially the only universally valid mechanism for certifying algorithmic stability. Since in practice, any test of stability would naturally be subject to computational constraints, exhaustive search is impossible and so this implies fundamental limits on our ability to test the stability property for a black-box algorithm.