Finite-Sample Two-Group Composite Hypothesis Testing via Machine Learning

TL;DR

This paper introduces a machine learning framework, especially using Deep Neural Networks, to construct powerful finite-sample hypothesis tests for two-group composite testing, applicable to adaptive clinical trials.

Contribution

It presents a novel automatic method leveraging machine learning to design hypothesis tests with improved power in finite samples, addressing challenges beyond exponential family models.

Findings

Demonstrates improved test power through simulations.

Shows applicability to adaptive clinical trial designs.

Provides an accessible implementation with R code and shiny app.

Abstract

In the problem of composite hypothesis testing, identifying the potential uniformly most powerful (UMP) unbiased test is of great interest. Beyond typical hypothesis settings with exponential family, it is usually challenging to prove the existence and further construct such UMP unbiased tests with finite sample size. For example in the COVID-19 pandemic with limited previous assumptions on the treatment for investigation and the standard of care, adaptive clinical trials are appealing due to ethical considerations, and the ability to accommodate uncertainty while conducting the trial. Although several methods have been proposed to control type I error rates, how to find a more powerful hypothesis testing strategy is still an open question. Motivated by this problem, we propose an automatic framework of constructing test statistics and corresponding critical values via machine learning methods to enhance power in a finite sample. In this article, we particularly illustrate the performance using Deep Neural Networks (DNN) and discuss its advantages. Simulations and two case studies of adaptive designs demonstrate that our method is automatic, general and pre-specified to construct statistics with satisfactory power in finite-sample. Supplemental materials are available online including R code and an R shiny app.