The CLC-UKET Dataset: Benchmarking Case Outcome Prediction for the UK Employment Tribunal

TL;DR

This paper introduces the CLC-UKET dataset, a large annotated collection of UK Employment Tribunal cases, and benchmarks case outcome prediction models, demonstrating that fine-tuned transformers outperform LLMs in this legal prediction task.

Contribution

The study creates a comprehensive, automatically annotated dataset for UKET cases and evaluates baseline models, advancing legal outcome prediction research.

Findings

Fine-tuned transformer models outperform zero-shot and few-shot LLMs.

Zero-shot LLM performance improves with task-related information.

The dataset enables benchmarking for employment dispute resolution.

Abstract

This paper explores the intersection of technological innovation and access to justice by developing a benchmark for predicting case outcomes in the UK Employment Tribunal (UKET). To address the challenge of extensive manual annotation, the study employs a large language model (LLM) for automatic annotation, resulting in the creation of the CLC-UKET dataset. The dataset consists of approximately 19,000 UKET cases and their metadata. Comprehensive legal annotations cover facts, claims, precedent references, statutory references, case outcomes, reasons and jurisdiction codes. Facilitated by the CLC-UKET data, we examine a multi-class case outcome prediction task in the UKET. Human predictions are collected to establish a performance reference for model comparison. Empirical results from baseline models indicate that finetuned transformer models outperform zero-shot and few-shot LLMs on the UKET prediction task. The performance of zero-shot LLMs can be enhanced by integrating task-related information into few-shot examples. We hope that the CLC-UKET dataset, along with human annotations and empirical findings, can serve as a valuable benchmark for employment-related dispute resolution.