Beyond Perfect Scores: Proof-by-Contradiction for Trustworthy Machine Learning

TL;DR

This paper proposes a stochastic proof-by-contradiction test to evaluate the trustworthiness of machine learning models in biomedical applications, distinguishing genuine causal learning from spurious correlations.

Contribution

It introduces a novel, interpretable trustworthiness testing method based on label permutation and p-values, applicable across diverse biomedical ML models.

Findings

Effective in identifying models relying on true clinical cues

Distinguishes genuine causal models from overfitting or shortcut learning

Provides interpretable statistical measures for trust assessment

Abstract

Machine learning (ML) models show strong promise for new biomedical prediction tasks, but concerns about trustworthiness have hindered their clinical adoption. In particular, it is often unclear whether a model relies on true clinical cues or on spurious hierarchical correlations in the data. This paper introduces a simple yet broadly applicable trustworthiness test grounded in stochastic proof-by-contradiction. Instead of just showing high test performance, our approach trains and tests on spurious labels carefully permuted based on a potential outcomes framework. A truly trustworthy model should fail under such label permutation; comparable accuracy across real and permuted labels indicates overfitting, shortcut learning, or data leakage. Our approach quantifies this behavior through interpretable Fisher-style p-values, which are well understood by domain experts across medical and life sciences. We evaluate our approach on multiple new bacterial diagnostics to separate tasks and models learning genuine causal relationships from those driven by dataset artifacts or statistical coincidences. Our work establishes a foundation to build rigor and trust between ML and life-science research communities, moving ML models one step closer to clinical adoption.