From BERT to T5: A Study of Named Entity Recognition

TL;DR

This study compares BERT and T5 models for named entity recognition, analyzing their performance, errors, and hyperparameters to inform future NLP applications.

Contribution

It provides a comparative analysis of encoder-only and sequence-to-sequence models for NER, including ablation studies and error analysis.

Findings

BERT and T5 show different strengths in NER performance.

Hyperparameter choices significantly affect model accuracy.

Error patterns reveal distinct challenges for each model.

Abstract

Named entity recognition (NER) has been one of the essential preliminary steps in modern NLP applications. This report focuses on implementing the NER task on finetuning two pretrained models: (i) an encoder-only model (BERT) with a simple classification head, and (ii) a sequence-to-sequence model (T5) with few-shot prompts. Under the original 7-class tag and 3-class simplified tag schemes, BERT is applied a weighted cross-entropy for training loss, and T5 is fine-tuned with two validation strategies. It also conducted an ablation study with different hyperparameters. Moreover, the related analysis provides valuable insights into common errors in BERT and the two models' performance. Based on a bunch of performance metrics, this report aims to compare the above two architectures and explore their abilities in the sequence labelling task, laying the groundwork for further practical use cases.