Robust Ultra-High-Dimensional Variable Selection With Correlated Structure Using Group Testing

TL;DR

This paper introduces a robust, multi-stage variable selection framework for high-dimensional genomic data that effectively handles correlated structures, contamination, and non-normality, improving biomarker discovery.

Contribution

It develops the Dorfman screening framework with robust variants, integrating hierarchical clustering, hypothesis testing, and elastic net, tailored for correlated and contaminated data.

Findings

Robust Dorfman methods outperform classical approaches under data contamination.

The framework achieves lower prediction errors in gene expression analysis.

Scales efficiently to ultra-high-dimensional genomic datasets.

Abstract

Background: High-dimensional genomic data exhibit strong group correlation structures that challenge conventional feature selection methods, which often assume feature independence or rely on pre-defined pathways and are sensitive to outliers and model misspecification. Methods: We propose the Dorfman screening framework, a multi-stage procedure that forms data-driven variable groups via hierarchical clustering, performs group and within-group hypothesis testing, and refines selection using elastic net or adaptive elastic net. Robust variants incorporate OGK-based covariance estimation, rank-based correlation, and Huber-weighted regression to handle contaminated and non-normal data. Results: In simulations, Dorfman-Sparse-Adaptive-EN performed best under normal conditions, while Robust-OGK-Dorfman-Adaptive-EN showed clear advantages under data contamination, outperforming classical Dorfman and competing methods. Applied to NSCLC gene expression data for trametinib response, robust Dorfman methods achieved the lowest prediction errors and enriched recovery of clinically relevant genes. Conclusions: The Dorfman framework provides an efficient and robust approach to genomic feature selection. Robust-OGK-Dorfman-Adaptive-EN offers strong performance under both ideal and contaminated conditions and scales to ultra-high-dimensional settings, making it well suited for modern genomic biomarker discovery.