How Good are LLMs at Relation Extraction under Low-Resource Scenario? Comprehensive Evaluation

TL;DR

This paper evaluates the effectiveness of large language models in relation extraction tasks within low-resource languages, using newly created multilingual datasets and empirical testing to identify performance gaps.

Contribution

It constructs and releases low-resource relation extraction datasets for 10 languages and empirically assesses LLM performance, highlighting challenges and potential areas for improvement.

Findings

LLMs perform poorly on low-resource RE tasks.

Data quality filtering improves dataset reliability.

Empirical results reveal significant performance gaps in LLMs for LRLs.

Abstract

Relation Extraction (RE) serves as a crucial technology for transforming unstructured text into structured information, especially within the framework of Knowledge Graph development. Its importance is emphasized by its essential role in various downstream tasks. Besides the conventional RE methods which are based on neural networks and pre-trained language models, large language models (LLMs) are also utilized in the research field of RE. However, on low-resource languages (LRLs), both conventional RE methods and LLM-based methods perform poorly on RE due to the data scarcity issues. To this end, this paper constructs low-resource relation extraction datasets in 10 LRLs in three regions (Central Asia, Southeast Asia and Middle East). The corpora are constructed by translating the original publicly available English RE datasets (NYT10, FewRel and CrossRE) using an effective multilingual machine translation. Then, we use the language perplexity (PPL) to filter out the low-quality data from the translated datasets. Finally, we conduct an empirical study and validate the performance of several open-source LLMs on these generated LRL RE datasets.