Combining Distantly Supervised Models with In Context Learning for Monolingual and Cross-Lingual Relation Extraction

TL;DR

This paper introduces HYDRE, a hybrid framework combining distantly supervised models and in-context learning with large language models to improve relation extraction in both monolingual and cross-lingual settings, especially for low-resource languages.

Contribution

HYDRE is a novel approach that integrates a trained DSRE model with dynamic exemplar retrieval and LLM prompting, enhancing relation extraction accuracy across languages.

Findings

HYDRE achieves up to 20 F1 point improvements in English.

HYDRE improves average F1 by 17 points on low-resource Indic languages.

Detailed ablations confirm HYDRE's effectiveness over other prompting methods.

Abstract

Distantly Supervised Relation Extraction (DSRE) remains a long-standing challenge in NLP, where models must learn from noisy bag-level annotations while making sentence-level predictions. While existing state-of-the-art (SoTA) DSRE models rely on task-specific training, their integration with in-context learning (ICL) using large language models (LLMs) remains underexplored. A key challenge is that the LLM may not learn relation semantics correctly, due to noisy annotation. In response, we propose HYDRE -- HYbrid Distantly Supervised Relation Extraction framework. It first uses a trained DSRE model to identify the top-k candidate relations for a given test sentence, then uses a novel dynamic exemplar retrieval strategy that extracts reliable, sentence-level exemplars from training data, which are then provided in LLM prompt for outputting the final relation(s). We further extend HYDRE to cross-lingual settings for RE in low-resource languages. Using available English DSRE training data, we evaluate all methods on English as well as a newly curated benchmark covering four diverse low-resource Indic languages -- Oriya, Santali, Manipuri, and Tulu. HYDRE achieves up to 20 F1 point gains in English and, on average, 17 F1 points on Indic languages over prior SoTA DSRE models. Detailed ablations exhibit HYDRE's efficacy compared to other prompting strategies.