RevDet: Robust and Memory Efficient Event Detection and Tracking in Large News Feeds

TL;DR

RevDet is a new algorithm for real-time, memory-efficient event detection and tracking in large, constantly updating news feeds, capable of accurately following event evolution over extended periods.

Contribution

It introduces RevDet, an iterative clustering method that tracks events with constant memory and includes a redundancy removal strategy, addressing challenges in large-scale news feed analysis.

Findings

Accurately recovers event chains in large datasets

Uses constant space for long-term event tracking

Outperforms standard clustering in memory efficiency

Abstract

With the ever-growing volume of online news feeds, event-based organization of news articles has many practical applications including better information navigation and the ability to view and analyze events as they develop. Automatically tracking the evolution of events in large news corpora still remains a challenging task, and the existing techniques for Event Detection and Tracking do not place a particular focus on tracking events in very large and constantly updating news feeds. Here, we propose a new method for robust and efficient event detection and tracking, which we call RevDet algorithm. RevDet adopts an iterative clustering approach for tracking events. Even though many events continue to develop for many days or even months, RevDet is able to detect and track those events while utilizing only a constant amount of space on main memory. We also devise a redundancy removal strategy which effectively eliminates duplicate news articles and substantially reduces the size of data. We construct a large, comprehensive new ground truth dataset specifically for event detection and tracking approaches by augmenting two existing datasets: w2e and GDELT. We implement RevDet algorithm and evaluate its performance on the ground truth event chains. We discover that our algorithm is able to accurately recover event chains in the ground-truth dataset. We also compare the memory efficiency of our algorithm with the standard single pass clustering approach, and demonstrate the appropriateness of our algorithm for event detection and tracking task in large news feeds.