Sequential monitoring using the Second Generation P-Value with Type I error controlled by monitoring frequency

TL;DR

This paper introduces SeqSGPV, a sequential monitoring method for scientific relevance that controls Type I error through monitoring frequency and affirmation, applicable to various hypotheses including PRISM, ROPE, and ROE.

Contribution

It formalizes sequential SGPV monitoring, introduces the PRISM hypothesis, and develops a novel error control method using monitoring frequency and affirmation steps.

Findings

SeqSGPV reduces wait time and sample size in simulations.

It effectively controls Type I error in various hypotheses.

Demonstrated applicability in real-world trial data.

Abstract

Many adaptive monitoring schemes adjust the required evidence toward a hypothesis to control Type I error. This shifts focus away from determining scientific relevance with an uncompromised degree of evidence. We propose sequentially monitoring the Second Generation P-Value (SGPV) on repeated intervals until establishing evidence for scientific relevance (SeqSGPV). SeqSGPV encompasses existing strategies to monitor Region of Practical Equivalence (ROPE) or Region of Equivalence (ROE) hypotheses. Hence, our focus is to formalize sequential SGPV monitoring; establish a novel set of scientific hypotheses, called PRISM, which is a ROE with a ROPE surrounding the null hypothesis; and use monitoring frequency and a novel affirmation step to control Type I error. Under immediate and delayed outcomes, we assess finite and limiting SeqSGPV operating characteristics when monitoring PRISM, ROPE, and null-bound ROE hypotheses. In extensive simulations, SeqSGPV PRISM monitoring reduced wait time for fully sequential monitoring, average sample size, and reversals of null hypothesis conclusions under the null. With real-world data, we design a SeqSGPV-monitored randomized trial. SeqSGPV is method-agnostic and easy to implement. Adjusting monitoring frequency/affirmation and monitoring a one-sided PRISM synergistically control Type I error. PRISM monitoring and adjusting monitoring frequency to control Type I error may have application beyond SeqSGPV.