Neural Machine Translation of Clinical Text: An Empirical Investigation into Multilingual Pre-Trained Language Models and Transfer-Learning

TL;DR

This study investigates neural machine translation for clinical texts using multilingual pre-trained models and transfer learning, achieving top performance and revealing that smaller models can outperform larger ones in clinical translation tasks.

Contribution

It demonstrates the effectiveness of transfer learning with multilingual pre-trained models in clinical translation and uncovers that smaller models can outperform larger models in this domain.

Findings

Small pre-trained models outperform larger models in clinical translation.

Transfer learning effectively adapts models to new languages like Spanish.

Models achieved top-level performance in ClinSpEn-2022 shared task.

Abstract

We conduct investigations on clinical text machine translation by examining multilingual neural network models using deep learning such as Transformer based structures. Furthermore, to address the language resource imbalance issue, we also carry out experiments using a transfer learning methodology based on massive multilingual pre-trained language models (MMPLMs). The experimental results on three subtasks including 1) clinical case (CC), 2) clinical terminology (CT), and 3) ontological concept (OC) show that our models achieved top-level performances in the ClinSpEn-2022 shared task on English-Spanish clinical domain data. Furthermore, our expert-based human evaluations demonstrate that the small-sized pre-trained language model (PLM) won over the other two extra-large language models by a large margin, in the clinical domain fine-tuning, which finding was never reported in the field. Finally, the transfer learning method works well in our experimental setting using the WMT21fb model to accommodate a new language space Spanish that was not seen at the pre-training stage within WMT21fb itself, which deserves more exploitation for clinical knowledge transformation, e.g. to investigate into more languages. These research findings can shed some light on domain-specific machine translation development, especially in clinical and healthcare fields. Further research projects can be carried out based on our work to improve healthcare text analytics and knowledge transformation. Our data will be openly available for research purposes at https://github.com/HECTA-UoM/ClinicalNMT