LegalLens Shared Task 2024: Legal Violation Identification in Unstructured Text

TL;DR

This paper reports on the LegalLens Shared Task 2024, which evaluated methods for identifying legal violations and their contexts in unstructured legal texts, highlighting the effectiveness of fine-tuned pre-trained language models.

Contribution

It introduces a shared task with a new dataset covering multiple legal domains and analyzes the performance of various approaches, emphasizing the success of fine-tuning pre-trained models.

Findings

Top models used fine-tuned pre-trained language models.

NER performance improved by 7.11% over baseline.

NLI performance improved by 5.7%, indicating room for further research.

Abstract

This paper presents the results of the LegalLens Shared Task, focusing on detecting legal violations within text in the wild across two sub-tasks: LegalLens-NER for identifying legal violation entities and LegalLens-NLI for associating these violations with relevant legal contexts and affected individuals. Using an enhanced LegalLens dataset covering labor, privacy, and consumer protection domains, 38 teams participated in the task. Our analysis reveals that while a mix of approaches was used, the top-performing teams in both tasks consistently relied on fine-tuning pre-trained language models, outperforming legal-specific models and few-shot methods. The top-performing team achieved a 7.11% improvement in NER over the baseline, while NLI saw a more marginal improvement of 5.7%. Despite these gains, the complexity of legal texts leaves room for further advancements.