Bilingual BSARD: Extending Statutory Article Retrieval to Dutch

TL;DR

This paper introduces bBSARD, a bilingual dataset for legal article retrieval in Dutch and French, and benchmarks various retrieval models, highlighting the competitiveness of traditional and fine-tuned models.

Contribution

The paper presents the first bilingual dataset for Belgian legal articles and provides comprehensive benchmarking of retrieval models in a multilingual legal context.

Findings

BM25 is a strong baseline in both languages.

Proprietary models outperform open models in zero-shot settings.

Fine-tuning small language-specific models can match or surpass proprietary models.

Abstract

Statutory article retrieval plays a crucial role in making legal information more accessible to both laypeople and legal professionals. Multilingual countries like Belgium present unique challenges for retrieval models due to the need for handling legal issues in multiple languages. Building on the Belgian Statutory Article Retrieval Dataset (BSARD) in French, we introduce the bilingual version of this dataset, bBSARD. The dataset contains parallel Belgian statutory articles in both French and Dutch, along with legal questions from BSARD and their Dutch translation. Using bBSARD, we conduct extensive benchmarking of retrieval models available for Dutch and French. Our benchmarking setup includes lexical models, zero-shot dense models, and fine-tuned small foundation models. Our experiments show that BM25 remains a competitive baseline compared to many zero-shot dense models in both languages. We also observe that while proprietary models outperform open alternatives in the zero-shot setting, they can be matched or surpassed by fine-tuning small language-specific models. Our dataset and evaluation code are publicly available.