An Evaluation Dataset for Legal Word Embedding: A Case Study On Chinese Codex

TL;DR

This paper introduces a legal analogy question set derived from Chinese Codex to evaluate legal word embeddings, revealing the prevalence of legal relations in the embeddings.

Contribution

It creates a novel legal analogy dataset (LARQS) for Chinese legal word embeddings and demonstrates its effectiveness for evaluation.

Findings

Legal relations are common in word embeddings.

LARQS effectively evaluates legal word embedding accuracy.

Legal analogy sets can reveal underlying relational structures.

Abstract

Word embedding is a modern distributed word representations approach widely used in many natural language processing tasks. Converting the vocabulary in a legal document into a word embedding model facilitates subjecting legal documents to machine learning, deep learning, and other algorithms and subsequently performing the downstream tasks of natural language processing vis-\`a-vis, for instance, document classification, contract review, and machine translation. The most common and practical approach of accuracy evaluation with the word embedding model uses a benchmark set with linguistic rules or the relationship between words to perform analogy reasoning via algebraic calculation. This paper proposes establishing a 1,134 Legal Analogical Reasoning Questions Set (LARQS) from the 2,388 Chinese Codex corpus using five kinds of legal relations, which are then used to evaluate the accuracy of the Chinese word embedding model. Moreover, we discovered that legal relations might be ubiquitous in the word embedding model.