The Massive Legal Embedding Benchmark (MLEB)

TL;DR

The paper introduces MLEB, the largest and most diverse open-source benchmark for legal information retrieval, covering multiple jurisdictions, document types, and tasks to advance legal AI research.

Contribution

It presents the creation of MLEB, a comprehensive benchmark with newly constructed datasets filling domain and jurisdiction gaps in legal IR.

Findings

Largest legal IR benchmark to date

Includes datasets from multiple jurisdictions and document types

Supports various tasks like search, classification, and QA

Abstract

We present the Massive Legal Embedding Benchmark (MLEB), the largest, most diverse, and most comprehensive open-source benchmark for legal information retrieval to date. MLEB consists of ten expert-annotated datasets spanning multiple jurisdictions (the US, UK, EU, Australia, Ireland, and Singapore), document types (cases, legislation, regulatory guidance, contracts, and literature), and task types (search, zero-shot classification, and question answering). Seven of the datasets in MLEB were newly constructed in order to fill domain and jurisdictional gaps in the open-source legal information retrieval landscape. We document our methodology in building MLEB and creating the new constituent datasets, and release our code, results, and data openly to assist with reproducible evaluations.