What's happening in your neighborhood? A Weakly Supervised Approach to Detect Local News

TL;DR

This paper presents a scalable, weakly supervised NLP pipeline for accurately detecting local news articles across multiple languages, improving upon rule-based methods by incorporating domain knowledge and auto data processing.

Contribution

The paper introduces a novel weakly supervised framework for local news detection that leverages domain knowledge and is scalable to multi-lingual settings, outperforming existing NER models.

Findings

Higher precision and recall than Stanford CoreNLP NER model

Effective in multi-lingual environments

Potential to improve local news recommendation accuracy

Abstract

Local news articles are a subset of news that impact users in a geographical area, such as a city, county, or state. Detecting local news (Step 1) and subsequently deciding its geographical location as well as radius of impact (Step 2) are two important steps towards accurate local news recommendation. Naive rule-based methods, such as detecting city names from the news title, tend to give erroneous results due to lack of understanding of the news content. Empowered by the latest development in natural language processing, we develop an integrated pipeline that enables automatic local news detection and content-based local news recommendations. In this paper, we focus on Step 1 of the pipeline, which highlights: (1) a weakly supervised framework incorporated with domain knowledge and auto data processing, and (2) scalability to multi-lingual settings. Compared with Stanford CoreNLP NER model, our pipeline has higher precision and recall evaluated on a real-world and human-labeled dataset. This pipeline has potential to more precise local news to users, helps local businesses get more exposure, and gives people more information about their neighborhood safety.