GerPS-Compare: Comparing NER methods for legal norm analysis

TL;DR

This paper compares rule-based, deep discriminative, and deep generative NER methods for legal norm texts in German, finding discriminative models outperform others in handling heterogeneous classes.

Contribution

It introduces a comparison of NER methods specifically for legal norms, highlighting the superior performance of deep discriminative models in this context.

Findings

Deep discriminative models outperform rule-based and generative models.

Rule-based and generative models perform similarly, with varying strengths across classes.

Heterogeneity of classes affects NER method performance.

Abstract

We apply NER to a particular sub-genre of legal texts in German: the genre of legal norms regulating administrative processes in public service administration. The analysis of such texts involves identifying stretches of text that instantiate one of ten classes identified by public service administration professionals. We investigate and compare three methods for performing Named Entity Recognition (NER) to detect these classes: a Rule-based system, deep discriminative models, and a deep generative model. Our results show that Deep Discriminative models outperform both the Rule-based system as well as the Deep Generative model, the latter two roughly performing equally well, outperforming each other in different classes. The main cause for this somewhat surprising result is arguably the fact that the classes used in the analysis are semantically and syntactically heterogeneous, in contrast to the classes used in more standard NER tasks. Deep Discriminative models appear to be better equipped for dealing with this heterogenerity than both generic LLMs and human linguists designing rule-based NER systems.