STCALIR: Semi-Synthetic Test Collection for Algerian Legal Information Retrieval
M'hamed Amine Hatem, Sofiane Batata, Amine Mammasse, Fai\c{c}al Azouaou

TL;DR
STCALIR is a framework that creates semi-synthetic legal test collections from raw documents, drastically reducing manual annotation effort while maintaining evaluation reliability.
Contribution
It introduces a semi-automated pipeline for generating legal test collections, significantly lowering manual annotation costs in low-resource domains.
Findings
Achieves 99% reduction in annotation workload.
Yields retrieval effectiveness comparable to human judgments.
System rankings strongly correlate with human evaluations.
Abstract
Test collections are essential for evaluating retrieval and re-ranking models. However, constructing such collections is challenging due to the high cost of manual annotation, particularly in specialized domains like Algerian legal texts, where high-quality corpora and relevance judgments are scarce. To address this limitation, we propose STCALIR, a framework for generating semi-synthetic test collections directly from raw legal documents. The pipeline follows the Cranfield paradigm, maintaining its core components of topics, corpus, and relevance judgments, while significantly reducing manual effort through automated multi-stage retrieval and filtering, achieving a 99% reduction in annotation workload. We validate STCALIR using the Mr. TyDi benchmark, demonstrating that the resulting semi-synthetic relevance judgments yield retrieval effectiveness comparable to human-annotated…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
