BERT got a Date: Introducing Transformers to Temporal Tagging
Satya Almasian, Dennis Aumiller, Michael Gertz

TL;DR
This paper introduces a transformer-based model for temporal expression tagging and classification, demonstrating that semi-supervised training with rule-based data improves performance, especially on rare classes.
Contribution
It identifies the best transformer architecture for joint temporal tagging and type classification and shows that semi-supervised training enhances accuracy over previous methods.
Findings
Transformer encoder-decoder with RoBERTa outperforms previous models.
Semi-supervised training with rule-based data improves rare class detection.
Model surpasses prior works in temporal tagging and classification.
Abstract
Temporal expressions in text play a significant role in language understanding and correctly identifying them is fundamental to various retrieval and natural language processing systems. Previous works have slowly shifted from rule-based to neural architectures, capable of tagging expressions with higher accuracy. However, neural models can not yet distinguish between different expression types at the same level as their rule-based counterparts. In this work, we aim to identify the most suitable transformer architecture for joint temporal tagging and type classification, as well as, investigating the effect of semi-supervised training on the performance of these systems. Based on our study of token classification variants and encoder-decoder architectures, we present a transformer encoder-decoder model using the RoBERTa language model as our best performing system. By supplementing…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNatural Language Processing Techniques · Topic Modeling · Speech and dialogue systems
MethodsRefunds@Expedia|||How do I get a full refund from Expedia? · Multi-Head Attention · Attention Is All You Need · Linear Layer · Attention Dropout · Weight Decay · Residual Connection · Linear Warmup With Linear Decay · Softmax · Dropout
