How Knowledge Graph and Attention Help? A Quantitative Analysis into Bag-level Relation Extraction

TL;DR

This paper quantitatively analyzes how knowledge graphs and attention mechanisms affect bag-level relation extraction, revealing nuanced effects and proposing a simple model variant that improves performance on real-world datasets.

Contribution

It introduces a dataset and a paradigm for quantitative evaluation of KG and attention in relation extraction, providing new insights into their effects and a simple improved model.

Findings

Higher attention accuracy may harm extraction performance.

Attention effectiveness is influenced by noise patterns in data.

KG enhances RE performance mainly through entity priors.

Abstract

Knowledge Graph (KG) and attention mechanism have been demonstrated effective in introducing and selecting useful information for weakly supervised methods. However, only qualitative analysis and ablation study are provided as evidence. In this paper, we contribute a dataset and propose a paradigm to quantitatively evaluate the effect of attention and KG on bag-level relation extraction (RE). We find that (1) higher attention accuracy may lead to worse performance as it may harm the model's ability to extract entity mention features; (2) the performance of attention is largely influenced by various noise distribution patterns, which is closely related to real-world datasets; (3) KG-enhanced attention indeed improves RE performance, while not through enhanced attention but by incorporating entity prior; and (4) attention mechanism may exacerbate the issue of insufficient training data. Based on these findings, we show that a straightforward variant of RE model can achieve significant improvements (6% AUC on average) on two real-world datasets as compared with three state-of-the-art baselines. Our codes and datasets are available at https://github.com/zig-kwin-hu/how-KG-ATT-help.