Modeling Between-Study Heterogeneity for Improved Reproducibility in Gene Signature Selection and Clinical Prediction

TL;DR

This paper introduces a novel statistical approach using penalized generalized linear mixed models to select gene signatures that are consistently non-zero across multiple studies, improving reproducibility and clinical relevance in genomic predictions.

Contribution

The paper presents a new method that accounts for between-study heterogeneity in gene signature selection, enhancing reproducibility over existing strategies that ignore such heterogeneity.

Findings

Method outperforms traditional approaches in simulations with heterogeneity.

Asymptotic results support the theoretical advantages of the proposed model.

Case study demonstrates practical utility in pancreatic cancer subtyping.

Abstract

In the genomic era, the identification of gene signatures associated with disease is of significant interest. Such signatures are often used to predict clinical outcomes in new patients and aid clinical decision-making. However, recent studies have shown that gene signatures are often not replicable. This occurrence has practical implications regarding the generalizability and clinical applicability of such signatures. To improve replicability, we introduce a novel approach to select gene signatures from multiple datasets whose effects are consistently non-zero and account for between-study heterogeneity. We build our model upon some rank-based quantities, facilitating integration over different genomic datasets. A high dimensional penalized Generalized Linear Mixed Model (pGLMM) is used to select gene signatures and address data heterogeneity. We compare our method to some commonly used strategies that select gene signatures ignoring between-study heterogeneity. We provide asymptotic results justifying the performance of our method and demonstrate its advantage in the presence of heterogeneity through thorough simulation studies. Lastly, we motivate our method through a case study subtyping pancreatic cancer patients from four gene expression studies.