Mortality Prediction Models with Clinical Notes Using Sparse Attention at the Word and Sentence Levels

TL;DR

This study explores sparse attention mechanisms at word and sentence levels in neural models for in-hospital mortality prediction from clinical notes, aiming to improve performance and transparency.

Contribution

It introduces and evaluates sparse attention mechanisms as an alternative to dense attention in clinical neural prediction models, demonstrating improved local attention performance.

Findings

Sparse attention outperforms dense attention at the word level.

Sparse attention focuses more on relevant directive words.

Sentence-level performance decreases due to sentence dropping.

Abstract

Intensive Care in-hospital mortality prediction has various clinical applications. Neural prediction models, especially when capitalising on clinical notes, have been put forward as improvement on currently existing models. However, to be acceptable these models should be performant and transparent. This work studies different attention mechanisms for clinical neural prediction models in terms of their discrimination and calibration. Specifically, we investigate sparse attention as an alternative to dense attention weights in the task of in-hospital mortality prediction from clinical notes. We evaluate the attention mechanisms based on: i) local self-attention over words in a sentence, and ii) global self-attention with a transformer architecture across sentences. We demonstrate that the sparse mechanism approach outperforms the dense one for the local self-attention in terms of predictive performance with a publicly available dataset, and puts higher attention to prespecified relevant directive words. The performance at the sentence level, however, deteriorates as sentences including the influential directive words tend to be dropped all together.