Mining Persistent Activity in Continually Evolving Networks

TL;DR

This paper introduces a novel approach to identify and analyze persistent activity patterns in continually evolving networks, revealing long-term regularities and anomalies in large-scale real-world data.

Contribution

It extends temporal motif analysis with a new persistence measure and presents PENminer, an efficient framework with algorithms for mining persistent activity snippets in evolving networks.

Findings

Discovered long-term regular activity in NYC taxi data during Hurricane Sandy.

Identified bursts of activity and anomalies that are not detectable by aggregate counts.

Outperformed baselines in anomaly detection with 9.8-48% higher AUC.

Abstract

Frequent pattern mining is a key area of study that gives insights into the structure and dynamics of evolving networks, such as social or road networks. However, not only does a network evolve, but often the way that it evolves, itself evolves. Thus, knowing, in addition to patterns' frequencies, for how long and how regularly they have occurred---i.e., their persistence---can add to our understanding of evolving networks. In this work, we propose the problem of mining activity that persists through time in continually evolving networks---i.e., activity that repeatedly and consistently occurs. We extend the notion of temporal motifs to capture activity among specific nodes, in what we call activity snippets, which are small sequences of edge-updates that reoccur. We propose axioms and properties that a measure of persistence should satisfy, and develop such a persistence measure. We also propose PENminer, an efficient framework for mining activity snippets' Persistence in Evolving Networks, and design both offline and streaming algorithms. We apply PENminer to numerous real, large-scale evolving networks and edge streams, and find activity that is surprisingly regular over a long period of time, but too infrequent to be discovered by aggregate count alone, and bursts of activity exposed by their lack of persistence. Our findings with PENminer include neighborhoods in NYC where taxi traffic persisted through Hurricane Sandy, the opening of new bike-stations, characteristics of social network users, and more. Moreover, we use PENminer towards identifying anomalies in multiple networks, outperforming baselines at identifying subtle anomalies by 9.8-48% in AUC.