Detecting Potential Topics In News Using BERT, CRF and Wikipedia

TL;DR

This paper presents a method for detecting important non-entity n-grams in news articles using BERT, Wikipedia data, and CRF, enhancing topic and hashtag identification for news recommendation systems.

Contribution

It introduces a novel approach combining Wikipedia, BERT, and CRF to identify key n-grams beyond traditional named entities in news texts.

Findings

Outperforms Flair, Spacy, Stanford NER in F1 and Recall

Effective in multilingual Indian news context

Detects important topic-related n-grams

Abstract

For a news content distribution platform like Dailyhunt, Named Entity Recognition is a pivotal task for building better user recommendation and notification algorithms. Apart from identifying names, locations, organisations from the news for 13+ Indian languages and use them in algorithms, we also need to identify n-grams which do not necessarily fit in the definition of Named-Entity, yet they are important. For example, "me too movement", "beef ban", "alwar mob lynching". In this exercise, given an English language text, we are trying to detect case-less n-grams which convey important information and can be used as topics and/or hashtags for a news. Model is built using Wikipedia titles data, private English news corpus and BERT-Multilingual pre-trained model, Bi-GRU and CRF architecture. It shows promising results when compared with industry best Flair, Spacy and Stanford-caseless-NER in terms of F1 and especially Recall.