# Structural modeling using overlapped group penalties for discovering predictive biomarkers for subgroup analysis

## TL;DR

This paper introduces a new penalized regression method with a novel penalty function to identify predictive biomarkers for subgroup analysis, ensuring hierarchical structure and applicability to various response types.

## Contribution

It proposes a generalized penalized regression with a novel penalty enforcing hierarchy between prognostic and predictive effects, and an efficient optimization algorithm named smog.

## Key findings

- Effective in selecting predictive biomarkers
- Accurate in subgroup identification
- Demonstrated superior performance in simulations and real data

## Abstract

The identification of predictive biomarkers from a large scale of covariates for subgroup analysis has attracted fundamental attention in medical research. In this article, we propose a generalized penalized regression method with a novel penalty function, for enforcing the hierarchy structure between the prognostic and predictive effects, such that a nonzero predictive effect must induce its ancestor prognostic effects being nonzero in the model. Our method is able to select useful predictive biomarkers by yielding a sparse, interpretable, and predictable model for subgroup analysis, and can deal with different types of response variable such as continuous, categorical, and time-to-event data. We show that our method is asymptotically consistent under some regularized conditions. To minimize the generalized penalized regression model, we propose a novel integrative optimization algorithm by integrating the majorization-minimization and the alternating direction method of multipliers, which is named after \texttt{smog}. The enriched simulation study and real case study demonstrate that our method is very powerful for discovering the true predictive biomarkers and identifying subgroups of patients.