Assessment of Generative Named Entity Recognition in the Era of Large Language Models

TL;DR

This paper systematically evaluates open-source large language models for generative named entity recognition, showing they can match or surpass traditional models with proper fine-tuning and formatting, and are not solely reliant on memorization.

Contribution

It provides a comprehensive assessment of open-source LLMs for generative NER, demonstrating their competitive performance and analyzing factors like output formats and memorization.

Findings

Open-source LLMs achieve competitive NER performance with fine-tuning.

Generative NER relies on instruction-following, not just memorization.

Fine-tuning minimally affects general capabilities, sometimes improving understanding.

Abstract

Named entity recognition (NER) is evolving from a sequence labeling task into a generative paradigm with the rise of large language models (LLMs). We conduct a systematic evaluation of open-source LLMs on both flat and nested NER tasks. We investigate several research questions including the performance gap between generative NER and traditional NER models, the impact of output formats, whether LLMs rely on memorization, and the preservation of general capabilities after fine-tuning. Through experiments across eight LLMs of varying scales and four standard NER datasets, we find that: (1) With parameter-efficient fine-tuning and structured formats like inline bracketed or XML, open-source LLMs achieve performance competitive with traditional encoder-based models and surpass closed-source LLMs like GPT-3; (2) The NER capability of LLMs stems from instruction-following and generative power, not mere memorization of entity-label pairs; and (3) Applying NER instruction tuning has minimal impact on general capabilities of LLMs, even improving performance on datasets like DROP due to enhanced entity understanding. These findings demonstrate that generative NER with LLMs is a promising, user-friendly alternative to traditional methods. We release the data and code at https://github.com/szu-tera/LLMs4NER.