Small Language Model Makes an Effective Long Text Extractor

TL;DR

This paper presents SeNER, a lightweight span-based NER method that effectively extracts long entity spans from extended texts using innovative attention mechanisms, achieving state-of-the-art accuracy while being GPU-memory efficient.

Contribution

Introduces SeNER, a novel span-based NER approach with bidirectional arrow attention and LogN-Scaling, reducing redundancy and improving long text entity extraction.

Findings

Achieves state-of-the-art accuracy on three long NER datasets.

Capable of extracting entities from long texts efficiently in GPU memory.

Outperforms existing span-based and generation-based methods.

Abstract

Named Entity Recognition (NER) is a fundamental problem in natural language processing (NLP). However, the task of extracting longer entity spans (e.g., awards) from extended texts (e.g., homepages) is barely explored. Current NER methods predominantly fall into two categories: span-based methods and generation-based methods. Span-based methods require the enumeration of all possible token-pair spans, followed by classification on each span, resulting in substantial redundant computations and excessive GPU memory usage. In contrast, generation-based methods involve prompting or fine-tuning large language models (LLMs) to adapt to downstream NER tasks. However, these methods struggle with the accurate generation of longer spans and often incur significant time costs for effective fine-tuning. To address these challenges, this paper introduces a lightweight span-based NER method called SeNER, which incorporates a bidirectional arrow attention mechanism coupled with LogN-Scaling on the [CLS] token to embed long texts effectively, and comprises a novel bidirectional sliding-window plus-shaped attention (BiSPA) mechanism to reduce redundant candidate token-pair spans significantly and model interactions between token-pair spans simultaneously. Extensive experiments demonstrate that our method achieves state-of-the-art extraction accuracy on three long NER datasets and is capable of extracting entities from long texts in a GPU-memory-friendly manner. Code: https://github.com/THUDM/scholar-profiling/tree/main/sener