Deep Learning of Semi-Competing Risk Data via a New Neural Expectation-Maximization Algorithm

TL;DR

This paper introduces a neural expectation-maximization algorithm for semi-competing risks in survival analysis, enabling improved prediction of disease progression and mortality in lung cancer patients using deep learning.

Contribution

It develops a novel neural EM algorithm that estimates baseline hazards, risk functions, and dependencies among transitions in semi-competing risks, bridging classical methods and deep learning.

Findings

Effective modeling of semi-competing risks in lung cancer data

Identification of key clinical and genetic predictors

Enhanced risk prediction accuracy

Abstract

Prognostication for lung cancer, a leading cause of mortality, remains a complex task, as it needs to quantify the associations of risk factors and health events spanning a patient's entire life. One challenge is that an individual's disease course involves non-terminal (e.g., disease progression) and terminal (e.g., death) events, which form semi-competing relationships. Our motivation comes from the Boston Lung Cancer Study, a large lung cancer survival cohort, which investigates how risk factors influence a patient's disease trajectory. Following developments in the prediction of time-to-event outcomes with neural networks, deep learning has become a focal area for the development of risk prediction methods in survival analysis. However, limited work has been done to predict multi-state or semi-competing risk outcomes, where a patient may experience adverse events such as disease progression prior to death. We propose a novel neural expectation-maximization algorithm to bridge the gap between classical statistical approaches and machine learning. Our algorithm enables estimation of the non-parametric baseline hazards of each state transition, risk functions of predictors, and the degree of dependence among different transitions, via a multi-task deep neural network with transition-specific sub-architectures. We apply our method to the Boston Lung Cancer Study and investigate the impact of clinical and genetic predictors on disease progression and mortality.