Leveraging sequence-to-sequence models for semantic annotation of Dutch pathology reports

TL;DR

This paper explores using AI models to automatically annotate Dutch pathology reports, showing good results for simpler reports but challenges with complex ones.

Contribution

The study introduces a T5-based model pre-trained on Dutch pathology data (PaTh5.NL) and evaluates constrained decoding for better annotation accuracy.

Findings

Fine-tuned PaTh5.NL models outperformed mT5 in shorter reports but struggled with complex texts.

Constrained decoding did not consistently improve patient retrieval despite higher BLEU scores.

Annotation quality declines with report complexity, especially in histology and autopsy reports.

Abstract

Palga Foundation is responsible for indexing Dutch pathology data across the Netherlands, which relies on annotations of pathology reports. These annotations, derived from the conclusion text, consist of codes from the Palga thesaurus, serving patient care and scientific research. However, manual annotation by pathologists is both labor-intensive and prone to errors. Therefore, in this study, we seek to leverage sequence-to-sequence transformer models, particularly Text-To-Text Transfer Transformer (T5)-based models, to generate these annotations. Additionally, we investigate a constrained decoding (CD) approach that encodes domain knowledge. We compare a standard multilingual T5 model (mT5) with our own T5 model (PaTh5.NL) pre-trained using Palga data with the goal of better aligning the model's learned representations with the specific structure, terminology, and annotation conventions used in Dutch pathology reports. We fine-tune both pre-trained models using default (DD) and CD and compare both decoding strategies. Performance is assessed using Bilingual Evaluation Understudy (BLEU) scores for quantitative evaluation and case-based evaluations for qualitative assessment, where we use the generated codes to retrieve patients from the Palga database. Quantitative evaluations indicated that our two fine-tuned PaTh5.NL models significantly outperformed the fine-tuned mT5 model, particularly for shorter histology and cytology reports, but performance of all models declined on longer or complex reports. The case-based evaluation revealed that, despite higher BLEU scores, the PaTh5.NL models did not consistently outperform the mT5 model in retrieving relevant patients. This study demonstrates that fine-tuned T5-based models can enhance the annotation process for Dutch pathology reports, though challenges remain regarding complex conclusion texts, especially in histology and autopsy reports. Future research should focus on expanding gold-standard datasets and developing post-processing algorithms to improve annotations' generalization.

Unlabelled Image

•Fine-tuned T5 transformer models can automatically annotate Dutch pathology reports.•Quality of the generated annotation is good for reports with short conclusions.•However, the quality decreases with report conclusion complexity.•Further refinement of the model scan be beneficiary for the annotation quality.

Fine-tuned T5 transformer models can automatically annotate Dutch pathology reports.

Quality of the generated annotation is good for reports with short conclusions.

However, the quality decreases with report conclusion complexity.

Further refinement of the model scan be beneficiary for the annotation quality.

Figures