Legal Retrieval for Public Defenders

TL;DR

This paper presents the NJ BriefBank, a legal retrieval tool designed to assist public defenders by surfacing relevant appellate briefs, and introduces a new benchmark dataset for evaluating legal retrieval systems in public defense.

Contribution

The paper develops a domain-specific legal retrieval system and releases a new benchmark dataset, addressing the gap in real-world public defense legal research support.

Findings

Adding domain knowledge improves retrieval quality.

Existing benchmarks do not transfer well to public defense research.

The benchmark correlates with expert-annotated datasets.

Abstract

AI tools are suggested as solutions to assist public agencies with heavy workloads. In public defense -- where a constitutional right to counsel meets the complexities of law, overwhelming caseloads, and constrained resources -- practitioners face especially taxing conditions. Yet, there is little evidence of how AI could meaningfully support defenders' day-to-day work. In partnership with the New Jersey Office of the Public Defender, we develop the NJ BriefBank, a retrieval tool which surfaces relevant appellate briefs to streamline legal research and writing. We show that existing retrieval benchmarks fail to transfer to real public defense research, however adding domain knowledge improves retrieval quality. This includes query expansion with legal reasoning, domain-specific data and curated synthetic examples. To facilitate further research, we release a taxonomy of realistic defender search queries and a manually annotated evaluation dataset for public defense retrieval. This benchmark is highly correlated with a proprietary retrieval dataset annotated by experienced public defenders. Our work improves on the status quo of realistic legal retrieval benchmarking and illustrates one approach to applying AI in a real-world public interest setting.