Automatic identification of diagnosis from hospital discharge letters via weakly-supervised Natural Language Processing

TL;DR

This paper introduces a weakly-supervised NLP pipeline that accurately classifies diagnoses from Italian discharge letters, significantly reducing manual annotation effort while maintaining high performance and adaptability across diseases.

Contribution

The study presents a novel weakly-supervised approach using clustering and transformer models to classify discharge letters without manual labels, improving scalability and efficiency.

Findings

Achieved an AUC of 77.68% and F1-score of 78.14% in classifying bronchiolitis.

Surpassed other unsupervised methods and approached supervised model performance.

Saved over 1,500 hours of expert annotation time for large datasets.

Abstract

Identifying patient diagnoses from discharge letters is essential to enable large-scale cohort selection and epidemiological research, but traditional supervised approaches rely on extensive manual annotation, which is often impractical for large textual datasets. In this study, we present a novel weakly-supervised Natural Language Processing pipeline designed to classify Italian discharge letters without requiring manual labelling. After extracting diagnosis-related sentences, the method leverages a transformer-based model with an additional pre-training on Italian medical documents to generate semantic embeddings. A two-level clustering procedure is applied to these embeddings, and the resulting clusters are mapped to the diseases of interest to derive weak labels for a subset of data, eventually used to train a transformer-based classifier. We evaluate the approach on a real-world case study on bronchiolitis in a corpus of 33,176 Italian discharge letters of children admitted to 44 emergency rooms or hospitals in the Veneto Region between 2017 and 2020. The pipeline achieves an area under the curve (AUC) of 77.68% ($\pm 4.30\%)$ and an F1-score of 78.14% ($\pm 4.89\%$) against manual annotations. Its performance surpasses other unsupervised methods and approaches fully supervised models, maintaining robustness to cluster selection and promising generalizability across different disease types. It allows saving approximately 3 minutes of expert time per discharge letter, resulting in more than 1,500 hours for a dataset like ours. This study demonstrates the feasibility of a weakly-supervised strategy for identifying diagnoses from Italian discharge letters. The pipeline achieves strong performance, is adaptable to various diseases, and offers a scalable solution for clinical text classification, reducing the need for manual annotation while maintaining reliable accuracy.