Stability Evaluation via Distributional Perturbation Analysis

TL;DR

This paper introduces a stability evaluation method for learning models based on distributional perturbations using optimal transport, addressing data corruptions and shifts, with practical convex formulations and empirical validation.

Contribution

We propose a novel stability evaluation criterion using optimal transport discrepancy with moment constraints, applicable to real-world distribution shifts and supported by tractable convex methods.

Findings

Effectively compares model stability under distribution shifts.

Guides strategies to improve model robustness.

Validated across multiple real-world applications.

Abstract

The performance of learning models often deteriorates when deployed in out-of-sample environments. To ensure reliable deployment, we propose a stability evaluation criterion based on distributional perturbations. Conceptually, our stability evaluation criterion is defined as the minimal perturbation required on our observed dataset to induce a prescribed deterioration in risk evaluation. In this paper, we utilize the optimal transport (OT) discrepancy with moment constraints on the \textit{(sample, density)} space to quantify this perturbation. Therefore, our stability evaluation criterion can address both \emph{data corruptions} and \emph{sub-population shifts} -- the two most common types of distribution shifts in real-world scenarios. To further realize practical benefits, we present a series of tractable convex formulations and computational methods tailored to different classes of loss functions. The key technical tool to achieve this is the strong duality theorem provided in this paper. Empirically, we validate the practical utility of our stability evaluation criterion across a host of real-world applications. These empirical studies showcase the criterion's ability not only to compare the stability of different learning models and features but also to provide valuable guidelines and strategies to further improve models.