Calibration of P-values for calibration and for deviation of a subpopulation from the full population

TL;DR

This paper develops and validates computational methods for calibrating P-values in significance tests, ensuring accurate interpretation of statistical results in assessing subpopulation deviations.

Contribution

It introduces efficient numerical techniques and rigorous proofs for P-value calibration, building on prior work on subpopulation analysis and significance testing.

Findings

Validated methods with open-source software

Achieved accurate P-value calibration

Demonstrated effectiveness through numerical examples

Abstract

The author's recent research papers, "Cumulative deviation of a subpopulation from the full population" and "A graphical method of cumulative differences between two subpopulations" (both published in volume 8 of Springer's open-access "Journal of Big Data" during 2021), propose graphical methods and summary statistics, without extensively calibrating formal significance tests. The summary metrics and methods can measure the calibration of probabilistic predictions and can assess differences in responses between a subpopulation and the full population while controlling for a covariate or score via conditioning on it. These recently published papers construct significance tests based on the scalar summary statistics, but only sketch how to calibrate the attained significance levels (also known as "P-values") for the tests. The present article reviews and synthesizes work spanning many decades in order to detail how to calibrate the P-values. The present paper presents computationally efficient, easily implemented numerical methods for evaluating properly calibrated P-values, together with rigorous mathematical proofs guaranteeing their accuracy, and illustrates and validates the methods with open-source software and numerical examples.