LegalDrill: Diagnosis-Driven Synthesis for Legal Reasoning in Small Language Models

TL;DR

LegalDrill is a novel framework that enhances small language models' legal reasoning by synthesizing and refining reasoning trajectories through a teacher-student approach, improving performance without requiring extensive expert annotations.

Contribution

It introduces a diagnosis-driven synthesis method that extracts and refines reasoning data from a capable teacher to improve small language models' legal reasoning abilities.

Findings

LegalDrill significantly improves SLM performance on legal benchmarks.

The approach reduces dependence on expensive expert annotations.

It enables scalable training of legal reasoning systems.

Abstract

Small language models (SLMs) are promising for real-world deployment due to their efficiency and low operational cost. However, their limited capacity struggles with high-stakes legal reasoning tasks that require coherent statute interpretation and logically consistent deduction. Furthermore, training SLMs for such tasks demands high-quality, concise reasoning trajectories, which are prohibitively expensive to manually collect and difficult to curate via standard rejection sampling, lacking granularity beyond final verdicts. To address these challenges, we propose {LegalDrill}, a diagnosis-driven synthesis framework that extracts and iteratively refines reasoning trajectories from a capable teacher via fine-grained prompting, then a self-reflective verification is employed to adaptively select the most effective data for the SLM student. The resulting data empower SLM training through supervised fine-tuning and direct preference optimization. Extensive experiments on several legal benchmarks demonstrate that {LegalDrill} significantly bolsters the legal reasoning capabilities of representative SLMs while bypassing the need for scarce expert annotations, paving a scalable path toward practical legal reasoning systems.