The Right Model for the Job: An Evaluation of Legal Multi-Label Classification Baselines

TL;DR

This paper evaluates various multi-label classification methods on legal datasets, highlighting the effectiveness of DistilRoBERTa, LegalBERT, T5, and CrossEncoder for legal document classification tasks.

Contribution

It provides a comprehensive comparison of traditional and Transformer-based MLC methods in the legal domain, analyzing their performance relative to dataset properties.

Findings

DistilRoBERTa and LegalBERT perform consistently well.

T5 offers comparable performance with generative advantages.

CrossEncoder can improve macro-F1 scores but at higher computational costs.

Abstract

Multi-Label Classification (MLC) is a common task in the legal domain, where more than one label may be assigned to a legal document. A wide range of methods can be applied, ranging from traditional ML approaches to the latest Transformer-based architectures. In this work, we perform an evaluation of different MLC methods using two public legal datasets, POSTURE50K and EURLEX57K. By varying the amount of training data and the number of labels, we explore the comparative advantage offered by different approaches in relation to the dataset properties. Our findings highlight DistilRoBERTa and LegalBERT as performing consistently well in legal MLC with reasonable computational demands. T5 also demonstrates comparable performance while offering advantages as a generative model in the presence of changing label sets. Finally, we show that the CrossEncoder exhibits potential for notable macro-F1 score improvements, albeit with increased computational costs.