Neural Contract Element Extraction Revisited: Letters from Sesame Street

TL;DR

This paper compares various neural models for contract element extraction, finding LSTM encoders outperform others, and emphasizes the importance of task-specific choices in model design.

Contribution

It provides a comprehensive evaluation of neural architectures and embeddings, highlighting the superiority of LSTMs and domain-specific embeddings for contract element extraction.

Findings

LSTM encoders outperform CNNs, Transformers, and BERT.

Domain-specific WORD2VEC embeddings are more effective than GloVe.

Recurrency in models is crucial for context-sensitive entity extraction.

Abstract

We investigate contract element extraction. We show that LSTM-based encoders perform better than dilated CNNs, Transformers, and BERT in this task. We also find that domain-specific WORD2VEC embeddings outperform generic pre-trained GLOVE embeddings. Morpho-syntactic features in the form of POS tag and token shape embeddings, as well as context-aware ELMO embeddings do not improve performance. Several of these observations contradict choices or findings of previous work on contract element extraction and generic sequence labeling tasks, indicating that contract element extraction requires careful task-specific choices. Analyzing the results of (i) plain TRANSFORMER-based and (ii) BERT-based models, we find that in the examined task, where the entities are highly context-sensitive, the lack of recurrency in TRANSFORMERs greatly affects their performance.