LEGAL-BERT: The Muppets straight out of Law School

TL;DR

This paper investigates how to best adapt BERT for legal NLP tasks, revealing that standard guidelines often fail and proposing a systematic approach with new models and hyper-parameter tuning.

Contribution

It introduces a comprehensive evaluation of BERT adaptation strategies in the legal domain and releases LEGAL-BERT models tailored for legal NLP applications.

Findings

Standard pre-training guidelines do not always generalize well to legal NLP.

Adapting BERT with additional pre-training on legal corpora improves performance.

The paper releases a family of LEGAL-BERT models for legal NLP tasks.

Abstract

BERT has achieved impressive performance in several NLP tasks. However, there has been limited investigation on its adaptation guidelines in specialised domains. Here we focus on the legal domain, where we explore several approaches for applying BERT models to downstream legal tasks, evaluating on multiple datasets. Our findings indicate that the previous guidelines for pre-training and fine-tuning, often blindly followed, do not always generalize well in the legal domain. Thus we propose a systematic investigation of the available strategies when applying BERT in specialised domains. These are: (a) use the original BERT out of the box, (b) adapt BERT by additional pre-training on domain-specific corpora, and (c) pre-train BERT from scratch on domain-specific corpora. We also propose a broader hyper-parameter search space when fine-tuning for downstream tasks and we release LEGAL-BERT, a family of BERT models intended to assist legal NLP research, computational law, and legal technology applications.