Complex networks for event detection in heterogeneous high volume news streams

TL;DR

This paper presents a real-time, network-based method for detecting important news events by analyzing named entity co-occurrences and community structures in high-volume news streams.

Contribution

It introduces a novel approach combining NLP and complex network analysis to identify significant news events through entity linking and community detection.

Findings

Weighted node degree tracking identifies change points indicating events

Community detection on KeyGraphs characterizes and distinguishes events

Prototype shows promising results in real-time event detection

Abstract

Detecting important events in high volume news streams is an important task for a variety of purposes.The volume and rate of online news increases the need for automated event detection methods thatcan operate in real time. In this paper we develop a network-based approach that makes the workingassumption that important news events always involve named entities (such as persons, locationsand organizations) that are linked in news articles. Our approach uses natural language processingtechniques to detect these entities in a stream of news articles and then creates a time-stamped seriesof networks in which the detected entities are linked by co-occurrence in articles and sentences. Inthis prototype, weighted node degree is tracked over time and change-point detection used to locateimportant events. Potential events are characterized and distinguished using community detectionon KeyGraphs that relate named entities and informative noun-phrases from related articles. Thismethodology already produces promising results and will be extended in future to include a widervariety of complex network analysis techniques.