Pat-DEVAL: Chain-of-Legal-Thought Evaluation for Patent Description

TL;DR

Pat-DEVAL is a novel multi-dimensional evaluation framework that uses a Chain-of-Legal-Thought reasoning process with large language models to assess patent descriptions for technical coherence and legal compliance.

Contribution

It introduces the first legally-constrained, multi-dimensional evaluation method for patent descriptions, significantly improving assessment accuracy over existing metrics.

Findings

Achieves a Pearson correlation of 0.69 with expert judgments.

Demonstrates a correlation of 0.73 in legal compliance evaluation.

Outperforms baseline metrics and existing LLM evaluators.

Abstract

Patent descriptions must deliver comprehensive technical disclosure while meeting strict legal standards such as enablement and written description requirements. Although large language models have enabled end-to-end automated patent drafting, existing evaluation approaches fail to assess long-form structural coherence and statutory compliance specific to descriptions. We propose Pat-DEVAL, the first multi-dimensional evaluation framework dedicated to patent description bodies. Leveraging the LLM-as-a-judge paradigm, Pat-DEVAL introduces Chain-of-Legal-Thought (CoLT), a legally-constrained reasoning mechanism that enforces sequential patent-law-specific analysis. Experiments validated by patent expert on our Pap2Pat-EvalGold dataset demonstrate that Pat-DEVAL achieves a Pearson correlation of 0.69, significantly outperforming baseline metrics and existing LLM evaluators. Notably, the framework exhibits a superior correlation of 0.73 in Legal-Professional Compliance, proving that the explicit injection of statutory constraints is essential for capturing nuanced legal validity. By establishing a new standard for ensuring both technical soundness and legal compliance, Pat-DEVAL provides a robust methodological foundation for the practical deployment of automated patent drafting systems.