Guarantees for Comprehensive Simulation Assessment of Statistical Methods

TL;DR

The paper introduces Continuous Simulation Extension (CSE), a framework that extends simulation-based evaluation of statistical methods to continuous parameter spaces, providing rigorous guarantees and calibration tools for complex models.

Contribution

CSE offers a novel, model-based approach to extend simulation results over parameter regions, enabling rigorous error control and calibration in adaptive and complex statistical procedures.

Findings

CSE provides valid confidence bands over parameter regions.

CSE successfully calibrates Type I Error control in Bayesian adaptive designs.

The method is applicable to models in exponential family and GLM forms.

Abstract

Simulation can evaluate a statistical method for properties such as Type I Error, FDR, or bias on a grid of hypothesized parameter values. But what about the gaps between the grid-points? Continuous Simulation Extension (CSE) is a proof-by-simulation framework which can supplement simulations with (1) confidence bands valid over regions of parameter space or (2) calibration of rejection thresholds to provide rigorous proof of strong Type I Error control. CSE extends simulation estimates at grid-points into bounds over nearby space using a model shift bound related to the Renyi divergence, which we analyze for models in exponential family or canonical GLM form. CSE can work with adaptive sampling, nuisance parameters, administrative censoring, multiple arms, multiple testing, Bayesian randomization, Bayesian decision-making, and inference algorithms of arbitrary complexity. As a case study, we calibrate for strong Type I Error control a Phase II/III Bayesian selection design with 4 unknown statistical parameters. Potential applications include calibration of new statistical procedures or streamlining regulatory review of adaptive trial designs. Our open-source software implementation imprint is available athttps://github.com/Confirm-Solutions/imprint