Zero- and Few-Shot Named-Entity Recognition: Case Study and Dataset in the Crime Domain (CrimeNER)

TL;DR

This paper introduces CrimeNER, a new dataset and case-study for zero- and few-shot NER in the crime domain, demonstrating the challenges and potential of current models on real-world crime data.

Contribution

It provides the first large-scale annotated crime NER dataset and evaluates zero- and few-shot NER performance using state-of-the-art models and large language models.

Findings

State-of-the-art models struggle with crime NER in low-resource settings.

Large language models show potential but have limitations in domain-specific NER.

CrimeNER dataset enables benchmarking and further research in crime-related information extraction.

Abstract

The extraction of critical information from crime-related documents is a crucial task for law enforcement agencies. Named-Entity Recognition (NER) can perform this task in extracting information about the crime, the criminal, or law enforcement agencies involved. However, there is a considerable lack of adequately annotated data on general real-world crime scenarios. To address this issue, we present CrimeNER, a case-study of Crime-related zero- and Few-Shot NER, and a general Crime-related Named-Entity Recognition database (CrimeNERdb) consisting of more than 1.5k annotated documents for the NER task extracted from public reports on terrorist attacks and the U.S. Department of Justice's press notes. We define 5 types of coarse crime entity and a total of 22 types of fine-grained entity. We address the quality of the case-study and the annotated data with experiments on Zero and Few-Shot settings with State-of-the-Art NER models as well as generalist and commonly used Large Language Models.