LegalRelectra: Mixed-domain Language Modeling for Long-range Legal Text Comprehension

TL;DR

LegalRelectra is a novel mixed-domain language model designed for long-range comprehension of legal and medical texts, outperforming general models on complex, mixed-domain legal documents.

Contribution

It introduces a mixed-domain legal-medical language model based on Electra and Reformer, enhancing long-range comprehension in specialized legal texts.

Findings

Improves processing of mixed-domain legal and medical texts

Enhances long-range text comprehension with Reformer architecture

Outperforms general and single-domain models on legal tasks

Abstract

The application of Natural Language Processing (NLP) to specialized domains, such as the law, has recently received a surge of interest. As many legal services rely on processing and analyzing large collections of documents, automating such tasks with NLP tools emerges as a key challenge. Many popular language models, such as BERT or RoBERTa, are general-purpose models, which have limitations on processing specialized legal terminology and syntax. In addition, legal documents may contain specialized vocabulary from other domains, such as medical terminology in personal injury text. Here, we propose LegalRelectra, a legal-domain language model that is trained on mixed-domain legal and medical corpora. We show that our model improves over general-domain and single-domain medical and legal language models when processing mixed-domain (personal injury) text. Our training architecture implements the Electra framework, but utilizes Reformer instead of BERT for its generator and discriminator. We show that this improves the model's performance on processing long passages and results in better long-range text comprehension.