Scalable Algorithms for Large Competing Risks Data

TL;DR

This paper introduces scalable algorithms for large competing risks data, including a new cyclic coordinate update method and a fast forward-backward scan, achieving over 1,000-fold speed improvements in sparse regression for the PSH model.

Contribution

It presents a novel cycBAR algorithm for accelerated sparse regression and a forward-backward scan method to reduce computational costs in the PSH model, enabling scalable analysis of large datasets.

Findings

CycBAR achieves significant speedups over traditional BAR.

Forward-backward scan reduces computation from O(n^2) to O(n).

Combined methods enable over 1,000-fold speed improvements.

Abstract

This paper develops two orthogonal contributions to scalable sparse regression for competing risks time-to-event data. First, we study and accelerate the broken adaptive ridge method (BAR), an $\ell_0$-based iteratively reweighted $\ell_2$-penalization algorithm that achieves sparsity in its limit, in the context of the Fine-Gray (1999) proportional subdistributional hazards (PSH) model. In particular, we derive a new algorithm for BAR regression, named cycBAR, that performs cyclic update of each coordinate using an explicit thresholding formula. The new cycBAR algorithm effectively avoids fitting multiple reweighted $\ell_2$-penalizations and thus yields impressive speedups over the original BAR algorithm. Second, we address a pivotal computational issue related to fitting the PSH model. Specifically, the computation costs of the log-pseudo likelihood and its derivatives for PSH model grow at the rate of $O(n^2)$ with the sample size $n$ in current implementations. We propose a novel forward-backward scan algorithm that reduces the computation costs to $O(n)$. The proposed method applies to both unpenalized and penalized estimation for the PSH model and has exhibited drastic speedups over current implementations. Finally, combining the two algorithms can yields $>1,000$ fold speedups over the original BAR algorithm. Illustrations of the impressive scalability of our proposed algorithm for large competing risks data are given using both simulations and a United States Renal Data System data.