Pretrained language model transfer on neural named entity recognition in Indonesian conversational texts

TL;DR

This paper demonstrates that transfer learning via language model pretraining significantly improves named entity recognition performance in Indonesian conversational texts, especially with limited labeled data.

Contribution

It introduces transfer learning methods for NER in Indonesian conversational texts, showing substantial performance gains with small datasets.

Findings

Both transfer learning variants outperform baseline models on small datasets.

Pretrained language models encode part-of-speech information useful for NER.

Achieved an absolute 32-point improvement in test F1 score with limited data.

Abstract

Named entity recognition (NER) is an important task in NLP, which is all the more challenging in conversational domain with their noisy facets. Moreover, conversational texts are often available in limited amount, making supervised tasks infeasible. To learn from small data, strong inductive biases are required. Previous work relied on hand-crafted features to encode these biases until transfer learning emerges. Here, we explore a transfer learning method, namely language model pretraining, on NER task in Indonesian conversational texts. We utilize large unlabeled data (generic domain) to be transferred to conversational texts, enabling supervised training on limited in-domain data. We report two transfer learning variants, namely supervised model fine-tuning and unsupervised pretrained LM fine-tuning. Our experiments show that both variants outperform baseline neural models when trained on small data (100 sentences), yielding an absolute improvement of 32 points of test F1 score. Furthermore, we find that the pretrained LM encodes part-of-speech information which is a strong predictor for NER.