Entity-Assisted Language Models for Identifying Check-worthy Sentences

TL;DR

This paper introduces a unified framework combining neural language models and knowledge graph entity embeddings to improve the identification of check-worthy sentences in political debates, outperforming traditional methods.

Contribution

The paper presents a novel entity-assisted framework that integrates semantic sentence analysis with entity embeddings, demonstrating superior performance over existing approaches.

Findings

Neural language models outperform TF.IDF and LSTM methods.

ALBERT model is the most effective among tested models.

Entity embeddings from knowledge graphs outperform similarity-based approaches.

Abstract

We propose a new uniform framework for text classification and ranking that can automate the process of identifying check-worthy sentences in political debates and speech transcripts. Our framework combines the semantic analysis of the sentences, with additional entity embeddings obtained through the identified entities within the sentences. In particular, we analyse the semantic meaning of each sentence using state-of-the-art neural language models such as BERT, ALBERT, and RoBERTa, while embeddings for entities are obtained from knowledge graph (KG) embedding models. Specifically, we instantiate our framework using five different language models, entity embeddings obtained from six different KG embedding models, as well as two combination methods leading to several Entity-Assisted neural language models. We extensively evaluate the effectiveness of our framework using two publicly available datasets from the CLEF' 2019 & 2020 CheckThat! Labs. Our results show that the neural language models significantly outperform traditional TF.IDF and LSTM methods. In addition, we show that the ALBERT model is consistently the most effective model among all the tested neural language models. Our entity embeddings significantly outperform other existing approaches from the literature that are based on similarity and relatedness scores between the entities in a sentence, when used alongside a KG embedding.