Towards Better Evaluation for Generated Patent Claims

TL;DR

This paper introduces Patent-CE, a comprehensive benchmark and PatClaimEval, a novel evaluation method, to improve the assessment of automatically generated patent claims, aligning automated metrics more closely with expert judgments.

Contribution

It presents the first detailed benchmark and evaluation method specifically designed for assessing patent claim quality, addressing the gap between automated metrics and human expert evaluations.

Findings

PatClaimEval correlates best with expert assessments across criteria.

Patent-CE provides detailed annotations for claim evaluation.

The proposed method improves accuracy of automated patent claim evaluation.

Abstract

Patent claims define the scope of protection and establish the legal boundaries of an invention. Drafting these claims is a complex and time-consuming process that usually requires the expertise of skilled patent attorneys, which can form a large access barrier for many small enterprises. To solve these challenges, researchers have investigated the use of large language models (LLMs) for automating patent claim generation. However, existing studies highlight inconsistencies between automated evaluation metrics and human expert assessments. To bridge this gap, we introduce Patent-CE, the first comprehensive benchmark for evaluating patent claims. Patent-CE includes comparative claim evaluations annotated by patent experts, focusing on five key criteria: feature completeness, conceptual clarity, terminology consistency, logical linkage, and overall quality. Additionally, we propose PatClaimEval, a novel multi-dimensional evaluation method specifically designed for patent claims. Our experiments demonstrate that PatClaimEval achieves the highest correlation with human expert evaluations across all assessment criteria among all tested metrics. This research provides the groundwork for more accurate evaluations of automated patent claim generation systems.