Hierarchical Label-wise Attention Transformer Model for Explainable ICD Coding

TL;DR

This paper introduces HiLAT, a hierarchical label-wise attention Transformer model that improves explainable ICD coding from clinical texts, outperforming previous models on MIMIC-III data with enhanced interpretability.

Contribution

The paper presents a novel hierarchical label-wise attention mechanism integrated with Transformers for explainable ICD coding, along with a new pretrained ClinicalplusXLNet model for better performance.

Findings

HiLAT+ClinicalplusXLNet achieves higher F1 scores than previous models.

Attention visualizations aid in explainability of ICD code predictions.

Model performs well on top-50 ICD-9 codes from MIMIC-III.

Abstract

International Classification of Diseases (ICD) coding plays an important role in systematically classifying morbidity and mortality data. In this study, we propose a hierarchical label-wise attention Transformer model (HiLAT) for the explainable prediction of ICD codes from clinical documents. HiLAT firstly fine-tunes a pretrained Transformer model to represent the tokens of clinical documents. We subsequently employ a two-level hierarchical label-wise attention mechanism that creates label-specific document representations. These representations are in turn used by a feed-forward neural network to predict whether a specific ICD code is assigned to the input clinical document of interest. We evaluate HiLAT using hospital discharge summaries and their corresponding ICD-9 codes from the MIMIC-III database. To investigate the performance of different types of Transformer models, we develop ClinicalplusXLNet, which conducts continual pretraining from XLNet-Base using all the MIMIC-III clinical notes. The experiment results show that the F1 scores of the HiLAT+ClinicalplusXLNet outperform the previous state-of-the-art models for the top-50 most frequent ICD-9 codes from MIMIC-III. Visualisations of attention weights present a potential explainability tool for checking the face validity of ICD code predictions.