Beyond P-Values: Importing Quantitative Finance's Risk and Regret Metrics for AI in Learning Health Systems

TL;DR

This paper proposes a new framework for evaluating AI in healthcare by importing risk and regret metrics from quantitative finance, addressing challenges of continual learning and non-stationary environments.

Contribution

It introduces a novel approach that reframes medical evidence for adaptive AI systems using risk-theoretic concepts, complementing traditional clinical trial methods.

Findings

Reframes evidence using calibration stability, risk bounds, and regret metrics.

Addresses risks specific to AI systems in dynamic clinical environments.

Provides a mathematical language for evaluating AI safety and performance.

Abstract

The increasing deployment of artificial intelligence (AI) in clinical settings challenges foundational assumptions underlying traditional frameworks of medical evidence. Classical statistical approaches, centered on randomized controlled trials, frequentist hypothesis testing, and static confidence intervals, were designed for fixed interventions evaluated under stable conditions. In contrast, AI-driven clinical systems learn continuously, adapt their behavior over time, and operate in non-stationary environments shaped by evolving populations, practices, and feedback effects. In such systems, clinical harm arises less from average error rates than from calibration drift, rare but severe failures, and the accumulation of suboptimal decisions over time. In this perspective, we argue that prevailing notions of statistical significance are insufficient for characterizing evidence and safety in learning health systems. Drawing on risk-theoretic concepts from quantitative finance and online decision theory, we propose reframing medical evidence for adaptive AI systems in terms of time-indexed calibration stability, bounded downside risk, and controlled cumulative regret. We emphasize that this approach does not replace randomized trials or causal inference, but complements them by addressing dimensions of risk and uncertainty that emerge only after deployment. This framework provides a principled mathematical language for evaluating AI-driven clinical systems under continual learning and offers implications for clinical practice, research design, and regulatory oversight.