Trust but Verify: Assigning Prediction Credibility by Counterfactual Constrained Learning

TL;DR

This paper introduces a novel, unsupervised method for assigning prediction credibility to machine learning models by analyzing the risk-fit trade-off through counterfactual constrained optimization, applicable across diverse architectures.

Contribution

It develops a model-agnostic, theoretically guaranteed framework for credibility assessment that does not require data or retraining, using duality theory to analyze the risk-fit trade-off.

Findings

Effective in data filtering and adversarial defense

Applicable to various neural network architectures

Provides theoretical guarantees for credibility measures

Abstract

Prediction credibility measures, in the form of confidence intervals or probability distributions, are fundamental in statistics and machine learning to characterize model robustness, detect out-of-distribution samples (outliers), and protect against adversarial attacks. To be effective, these measures should (i) account for the wide variety of models used in practice, (ii) be computable for trained models or at least avoid modifying established training procedures, (iii) forgo the use of data, which can expose them to the same robustness issues and attacks as the underlying model, and (iv) be followed by theoretical guarantees. These principles underly the framework developed in this work, which expresses the credibility as a risk-fit trade-off, i.e., a compromise between how much can fit be improved by perturbing the model input and the magnitude of this perturbation (risk). Using a constrained optimization formulation and duality theory, we analyze this compromise and show that this balance can be determined counterfactually, without having to test multiple perturbations. This results in an unsupervised, a posteriori method of assigning prediction credibility for any (possibly non-convex) differentiable model, from RKHS-based solutions to any architecture of (feedforward, convolutional, graph) neural network. Its use is illustrated in data filtering and defense against adversarial attacks.