Enriching language models with graph-based context information to better understand textual data

TL;DR

This paper explores integrating graph-based contextual information into BERT to improve text understanding, demonstrating a modest performance boost on a classification task with minimal parameter increase.

Contribution

It introduces a method to incorporate graph-derived context into BERT, showing improved classification accuracy on Pubmed dataset.

Findings

Error reduced from 8.51% to 7.96% on Pubmed dataset

Model complexity increased by only 1.6%

Graph-based context enhances language model performance

Abstract

A considerable number of texts encountered daily are somehow connected with each other. For example, Wikipedia articles refer to other articles via hyperlinks, scientific papers relate to others via citations or (co)authors, while tweets relate via users that follow each other or reshare content. Hence, a graph-like structure can represent existing connections and be seen as capturing the "context" of the texts. The question thus arises if extracting and integrating such context information into a language model might help facilitate a better automated understanding of the text. In this study, we experimentally demonstrate that incorporating graph-based contextualization into BERT model enhances its performance on an example of a classification task. Specifically, on Pubmed dataset, we observed a reduction in error from 8.51% to 7.96%, while increasing the number of parameters just by 1.6%. Our source code: https://github.com/tryptofanik/gc-bert