Distance Assisted Recursive Testing

TL;DR

DART is a novel recursive testing framework that leverages known feature distances to improve feature selection accuracy, demonstrated through theoretical analysis, simulations, and a clinical microbiota study.

Contribution

It introduces a two-stage method transforming distance matrices into aggregation trees for enhanced multiple testing in feature selection.

Findings

DART controls false discovery proportion effectively in high probability.

DART outperforms existing methods in simulations under various models.

Application to clinical data identified microbiota impacted by treatment.

Abstract

In many applications, a large number of features are collected with the goal to identify a few important ones. Sometimes, these features lie in a metric space with a known distance matrix, which partially reflects their co-importance pattern. Proper use of the distance matrix will boost the power of identifying important features. Hence, we develop a new multiple testing framework named the Distance Assisted Recursive Testing (DART). DART has two stages. In stage 1, we transform the distance matrix into an aggregation tree, where each node represents a set of features. In stage 2, based on the aggregation tree, we set up dynamic node hypotheses and perform multiple testing on the tree. All rejections are mapped back to the features. Under mild assumptions, the false discovery proportion of DART converges to the desired level in high probability converging to one. We illustrate by theory and simulations that DART has superior performance under various models compared to the existing methods. We applied DART to a clinical trial in the allogeneic stem cell transplantation study to identify the gut microbiota whose abundance will be impacted by the after-transplant care.