Building a Japanese Document-Level Relation Extraction Dataset Assisted by Cross-Lingual Transfer

TL;DR

This paper explores leveraging English DocRE datasets for Japanese by transfer learning, constructing a Japanese dataset, analyzing errors, and proposing a model-assisted annotation method that reduces human effort.

Contribution

It introduces a Japanese DocRE dataset created via cross-lingual transfer and demonstrates how model-assisted annotation can improve efficiency.

Findings

Transferred datasets improve initial relation predictions.

Model assistance reduces human annotation steps by 50%.

Japanese DocRE presents unique challenges for cross-lingual transfer.

Abstract

Document-level Relation Extraction (DocRE) is the task of extracting all semantic relationships from a document. While studies have been conducted on English DocRE, limited attention has been given to DocRE in non-English languages. This work delves into effectively utilizing existing English resources to promote DocRE studies in non-English languages, with Japanese as the representative case. As an initial attempt, we construct a dataset by transferring an English dataset to Japanese. However, models trained on such a dataset suffer from low recalls. We investigate the error cases and attribute the failure to different surface structures and semantics of documents translated from English and those written by native speakers. We thus switch to explore if the transferred dataset can assist human annotation on Japanese documents. In our proposal, annotators edit relation predictions from a model trained on the transferred dataset. Quantitative analysis shows that relation recommendations suggested by the model help reduce approximately 50% of the human edit steps compared with the previous approach. Experiments quantify the performance of existing DocRE models on our collected dataset, portraying the challenges of Japanese and cross-lingual DocRE.