Measuring the Robustness of Predictive Probability for Early Stopping in Experimental Design

TL;DR

This paper evaluates the robustness of predictive probability for early stopping in Bayesian adaptive experimental designs, particularly for continuous data in reliability testing, demonstrating potential for resource savings with minimal assumption violations.

Contribution

It extends the application of predictive probability for early stopping to continuous data in reliability experiments, providing robustness insights and practical benefits.

Findings

Early stopping based on PP is robust to minor assumption violations.

Using PP can save approximately 33% of experimental runs.

Simulation confirms effectiveness in reliability certification tasks.

Abstract

Physical experiments in the national security domain are often expensive and time-consuming. Test engineers must certify the compatibility of aircraft and their weapon systems before they can be deployed in the field, but the testing required is time consuming, expensive, and resource limited. Adopting Bayesian adaptive designs are a promising way to borrow from the successes seen in the clinical trials domain. The use of predictive probability (PP) to stop testing early and make faster decisions is particularly appealing given the aforementioned constraints. Given the high-consequence nature of the tests performed in the national security space, a strong understanding of new methods is required before being deployed. Although PP has been thoroughly studied for binary data, there is less work with continuous data, which often in reliability studies interested in certifying the specification limits of components. A simulation study evaluating the robustness of this approach indicate early stopping based on PP is reasonably robust to minor assumption violations, especially when only a few interim analyses are conducted. A post-hoc analysis exploring whether release requirements of a weapon system from an aircraft are within specification with desired reliability resulted in stopping the experiment early and saving 33% of the experimental runs.