Learning ULMFiT and Self-Distillation with Calibration for Medical Dialogue System

TL;DR

This paper enhances medical dialogue systems by calibrating ULMFiT and self-distillation models using label smoothing and temperature scaling, leading to improved accuracy and trustworthiness in healthcare NLP applications.

Contribution

It introduces a calibration approach for ULMFiT and self-distillation in medical dialogue systems using label smoothing and temperature scaling, improving model reliability.

Findings

Calibrated models outperform conventional methods in accuracy.

Temperature scaling improves confidence estimation.

Calibration enhances robustness in medical NLP tasks.

Abstract

A medical dialogue system is essential for healthcare service as providing primary clinical advice and diagnoses. It has been gradually adopted and practiced in medical organizations in the form of a conversational bot, largely due to the advancement of NLP. In recent years, the introduction of state-of-the-art deep learning models and transfer learning techniques like Universal Language Model Fine Tuning (ULMFiT) and Knowledge Distillation (KD) largely contributes to the performance of NLP tasks. However, some deep neural networks are poorly calibrated and wrongly estimate the uncertainty. Hence the model is not trustworthy, especially in sensitive medical decision-making systems and safety tasks. In this paper, we investigate the well-calibrated model for ULMFiT and self-distillation (SD) in a medical dialogue system. The calibrated ULMFiT (CULMFiT) is obtained by incorporating label smoothing (LS), a commonly used regularization technique to achieve a well-calibrated model. Moreover, we apply the technique to recalibrate the confidence score called temperature scaling (TS) with KD to observe its correlation with network calibration. To further understand the relation between SD and calibration, we use both fixed and optimal temperatures to fine-tune the whole model. All experiments are conducted on the consultation backpain dataset collected by experts then further validated using a large publicly medial dialogue corpus. We empirically show that our proposed methodologies outperform conventional methods in terms of accuracy and robustness.