Robustly Pre-trained Neural Model for Direct Temporal Relation Extraction

TL;DR

This paper demonstrates that large pre-trained language models like RoBERTa significantly improve the accuracy of extracting temporal relations in clinical texts, surpassing previous SVM-based methods.

Contribution

The study introduces the application of advanced pre-trained models, especially RoBERTa, for clinical temporal relation extraction, showing substantial performance gains over prior approaches.

Findings

RoBERTa improves F measure by 0.0864 over previous state-of-the-art.

Pre-training on larger corpora enhances model performance.

Neural models outperform traditional SVM approaches in clinical temporal relation tasks.

Abstract

Background: Identifying relationships between clinical events and temporal expressions is a key challenge in meaningfully analyzing clinical text for use in advanced AI applications. While previous studies exist, the state-of-the-art performance has significant room for improvement. Methods: We studied several variants of BERT (Bidirectional Encoder Representations using Transformers) some involving clinical domain customization and the others involving improved architecture and/or training strategies. We evaluated these methods using a direct temporal relations dataset which is a semantically focused subset of the 2012 i2b2 temporal relations challenge dataset. Results: Our results show that RoBERTa, which employs better pre-training strategies including using 10x larger corpus, has improved overall F measure by 0.0864 absolute score (on the 1.00 scale) and thus reducing the error rate by 24% relative to the previous state-of-the-art performance achieved with an SVM (support vector machine) model. Conclusion: Modern contextual language modeling neural networks, pre-trained on a large corpus, achieve impressive performance even on highly-nuanced clinical temporal relation tasks.