IPEval: A Bilingual Intellectual Property Agency Consultation Evaluation Benchmark for Large Language Models

TL;DR

IPEval is a comprehensive benchmark designed to evaluate large language models' understanding and reasoning in intellectual property law, covering creation, application, protection, and management across multiple IP types in English and Chinese.

Contribution

This paper introduces IPEval, the first dedicated evaluation benchmark for LLMs in the IP domain, with diverse questions and evaluation methods to assess legal reasoning and knowledge.

Findings

English LLMs like GPT perform best in English IP tasks.

Chinese LLMs excel in Chinese IP questions.

Specialized IP LLMs lag behind general-purpose models.

Abstract

The rapid development of Large Language Models (LLMs) in vertical domains, including intellectual property (IP), lacks a specific evaluation benchmark for assessing their understanding, application, and reasoning abilities. To fill this gap, we introduce IPEval, the first evaluation benchmark tailored for IP agency and consulting tasks. IPEval comprises 2657 multiple-choice questions across four major dimensions: creation, application, protection, and management of IP. These questions span patent rights (inventions, utility models, designs), trademarks, copyrights, trade secrets, and other related laws. Evaluation methods include zero-shot, 5-few-shot, and Chain of Thought (CoT) for seven LLM types, predominantly in English or Chinese. Results show superior English performance by models like GPT series and Qwen series, while Chinese-centric LLMs excel in Chinese tests, albeit specialized IP LLMs lag behind general-purpose ones. Regional and temporal aspects of IP underscore the need for LLMs to grasp legal nuances and evolving laws. IPEval aims to accurately gauge LLM capabilities in IP and spur development of specialized models. Website: \url{https://ipeval.github.io/}