Towards Realistic Low-resource Relation Extraction: A Benchmark with Empirical Baseline Study

TL;DR

This paper conducts an empirical study on low-resource relation extraction using pre-trained language models, evaluating prompt methods, balancing techniques, and data augmentation across a diverse benchmark of 8 datasets.

Contribution

It introduces a comprehensive benchmark for low-resource relation extraction and systematically compares various schemes to identify effective strategies.

Findings

Prompt-based tuning shows potential but needs improvement.

Balancing methods are not always effective for long-tailed data.

Data augmentation significantly improves performance, self-training less so.

Abstract

This paper presents an empirical study to build relation extraction systems in low-resource settings. Based upon recent pre-trained language models, we comprehensively investigate three schemes to evaluate the performance in low-resource settings: (i) different types of prompt-based methods with few-shot labeled data; (ii) diverse balancing methods to address the long-tailed distribution issue; (iii) data augmentation technologies and self-training to generate more labeled in-domain data. We create a benchmark with 8 relation extraction (RE) datasets covering different languages, domains and contexts and perform extensive comparisons over the proposed schemes with combinations. Our experiments illustrate: (i) Though prompt-based tuning is beneficial in low-resource RE, there is still much potential for improvement, especially in extracting relations from cross-sentence contexts with multiple relational triples; (ii) Balancing methods are not always helpful for RE with long-tailed distribution; (iii) Data augmentation complements existing baselines and can bring much performance gain, while self-training may not consistently achieve advancement to low-resource RE. Code and datasets are in https://github.com/zjunlp/LREBench.