Ontology-Based Medication Named Entity Recognition Using Pretrained Transformer Models From a Thai Hospital: Model Fine-Tuning and Validation Study

TL;DR

This study fine-tunes transformer models to extract medication information from Thai hospital records, achieving high accuracy with ClinicalBERT.

Contribution

The novel contribution is the application of ontology-based annotation with transformer models for medication NER in Thai clinical text.

Findings

ClinicalBERT achieved the highest exact F1-score of 0.973 in medication entity recognition.

All models struggled most with identifying 'Unit of Measure' entities due to implicit information in the text.

Ontology-based NER using transformer models shows promise for improving data standardization in Thai healthcare.

Abstract

Extracting accurate medication information from Thai hospital records presents challenges due to the narrative style of medical notes, which often combine Thai and English terminology. Named entity recognition (NER) serves as the foundational step for advanced clinical information extraction (IE) tasks, including medical concept normalization and relation extraction. This study aimed to establish a robust NER framework to address these difficulties by leveraging ontology-based annotation and pretrained transformer models.

The primary objective of this study was to evaluate the performance of 5 fine-tuned pretrained transformer models—BioClinicalBERT, ClinicalBERT, PubMedBERT, MultilingualBERT, and ThaiBERT—based on Bidirectional Encoder Representations from Transformers (BERT) in extracting structured medication information from unstructured Thai hospital discharge summaries.

Ninety discharge summaries were collected from Maharaj Nakhon Chiang Mai Hospital. These documents were annotated by physicians following the annotation guidelines based on international standards, including Systematized Nomenclature of Medicine-Clinical Terms (SNOMED-CT) and Health Level Seven Fast Healthcare Interoperability Resources (HL7 FHIR). The dataset was divided into fine-tuning (70 records, 78%, 2030 annotated spans), validation (10 records, 11%, 277 annotated spans), and testing sets (10 records, 11%, 358 annotated spans). The 5 transformer models were fine-tuned and evaluated using this annotated data to recognize and classify key medication entities (substance, route of administration, unit of measure, time patterns, and unit of presentation).

We found that all models had good NER performance metrics in both the validation and test datasets. Regarding test performance, ClinicalBERT achieved the highest exact F1-score at 0.973, compared with 0.968 for BioClinicalBERT, 0.925 for PubMedBERT, 0.931 for MultilingualBERT, and 0.969 for ThaiBERT. All models showed strength in accurately identifying “Substance” and “Dosage” entities, whereas “Unit of Measure” proved to be the most challenging entity type due to implicit information in the source text for all models.

The findings suggest that ontology-based medication IE using transformer-based models holds promise for enhancing data standardization and interoperability within the Thai health care system. Future work will need to leverage the granular annotations preserved in the dataset to develop medical concept normalization and relation extraction models to complete the medical IE system.

Figures