Does constituency analysis enhance domain-specific pre-trained BERT models for relation extraction?

TL;DR

This study investigates whether incorporating constituency-based syntactic information into domain-specific BERT models improves relation extraction performance, finding it increases precision but reduces recall in a biomedical context.

Contribution

It demonstrates the impact of syntactic information infusion into BERT models on relation extraction, highlighting trade-offs in precision and recall.

Findings

Adding syntactic info improves precision

Syntactic info decreases recall for rare relations

Ensemble of bioBERT, sciBERT, and const-bioBERT used for relation extraction

Abstract

Recently many studies have been conducted on the topic of relation extraction. The DrugProt track at BioCreative VII provides a manually-annotated corpus for the purpose of the development and evaluation of relation extraction systems, in which interactions between chemicals and genes are studied. We describe the ensemble system that we used for our submission, which combines predictions of fine-tuned bioBERT, sciBERT and const-bioBERT models by majority voting. We specifically tested the contribution of syntactic information to relation extraction with BERT. We observed that adding constituentbased syntactic information to BERT improved precision, but decreased recall, since relations rarely seen in the train set were less likely to be predicted by BERT models in which the syntactic information is infused. Our code is available online [https://github.com/Maple177/drugprot-relation-extraction].