Bayesian Dynamic Modeling and Monitoring of Network Flows

TL;DR

This paper introduces Bayesian dynamic models for real-time monitoring of large-scale network flows, enabling adaptive analysis and anomaly detection in complex e-commerce web traffic data.

Contribution

It develops scalable Bayesian dynamic generalized linear models for multivariate network flows, integrating decouple/recouple techniques for efficient large-scale analysis.

Findings

Successfully modeled over 56,000 node pairs in web traffic data.

Demonstrated effective detection of flow changes and anomalies.

Provided a flexible framework applicable to various dynamic network studies.

Abstract

In the context of a motivating study of dynamic network flow data on a large-scale e-commerce web site, we develop Bayesian models for on-line/sequential analysis for monitoring and adapting to changes reflected in node-node traffic. For large-scale networks, we customize core Bayesian time series analysis methods using dynamic generalized linear models (DGLMs). These are integrated into the context of multivariate networks using the concept of decouple/recouple that was recently introduced in multivariate time series. This method enables flexible dynamic modeling of flows on large-scale networks and exploitation of partial parallelization of analysis while maintaining coherence with an over-arching multivariate dynamic flow model. This approach is anchored in a case-study on internet data, with flows of visitors to a commercial news web site defining a long time series of node-node counts on over 56,000 node pairs. Central questions include characterizing inherent stochasticity in traffic patterns, understanding node-node interactions, adapting to dynamic changes in flows and allowing for sensitive monitoring to flag anomalies. The methodology of dynamic network DGLMs applies to many dynamic network flow studies.