FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processing

TL;DR

This paper introduces FairLex, a comprehensive multilingual benchmark suite for assessing fairness in legal NLP models across various jurisdictions, languages, and attributes, revealing persistent disparities and challenges in achieving fairness.

Contribution

The paper presents a new multilingual benchmark suite for evaluating fairness in legal NLP, covering multiple jurisdictions, languages, and attributes, and analyzes the effectiveness of existing fairness techniques.

Findings

Group disparities are prevalent across models and settings.

Current fairness techniques do not guarantee or consistently improve fairness.

Open challenges remain in developing robust fairness methods for legal NLP.

Abstract

We present a benchmark suite of four datasets for evaluating the fairness of pre-trained language models and the techniques used to fine-tune them for downstream tasks. Our benchmarks cover four jurisdictions (European Council, USA, Switzerland, and China), five languages (English, German, French, Italian and Chinese) and fairness across five attributes (gender, age, region, language, and legal area). In our experiments, we evaluate pre-trained language models using several group-robust fine-tuning techniques and show that performance group disparities are vibrant in many cases, while none of these techniques guarantee fairness, nor consistently mitigate group disparities. Furthermore, we provide a quantitative and qualitative analysis of our results, highlighting open challenges in the development of robustness methods in legal NLP.