Applications of BERT Based Sequence Tagging Models on Chinese Medical Text Attributes Extraction

TL;DR

This paper explores various BERT-based sequence tagging models, including LSTM-CRF, CNN, UCNN, WaveNet, and SelfAttention, for extracting attributes from Chinese medical texts, achieving competitive results through model ensembling.

Contribution

It introduces the application of multiple BERT-based sequence models to Chinese medical text attribute extraction and demonstrates the effectiveness of ensembling these models.

Findings

Models reach similar performance levels.

Ensembling improves overall system accuracy.

Achieved strong results on CCKS 2019 task 1.

Abstract

We convert the Chinese medical text attributes extraction task into a sequence tagging or machine reading comprehension task. Based on BERT pre-trained models, we have not only tried the widely used LSTM-CRF sequence tagging model, but also other sequence models, such as CNN, UCNN, WaveNet, SelfAttention, etc, which reaches similar performance as LSTM+CRF. This sheds a light on the traditional sequence tagging models. Since the aspect of emphasis for different sequence tagging models varies substantially, ensembling these models adds diversity to the final system. By doing so, our system achieves good performance on the task of Chinese medical text attributes extraction (subtask 2 of CCKS 2019 task 1).