Efficient Online Inference for Infinite Evolutionary Cluster models with Applications to Latent Social Event Discovery

TL;DR

This paper introduces a scalable, online inference method for the Recurrent Chinese Restaurant Process, enabling effective social event discovery from large-scale social media data by overcoming non-conjugacy limitations.

Contribution

It presents a novel solution to non-conjugacy in RCRP using Sequential Monte Carlo methods, significantly improving scalability and predictive performance.

Findings

Scalable inference on tens of millions of documents.

High-quality social event detection with interpretable clusters.

Better predictive accuracy than prior methods.

Abstract

The Recurrent Chinese Restaurant Process (RCRP) is a powerful statistical method for modeling evolving clusters in large scale social media data. With the RCRP, one can allow both the number of clusters and the cluster parameters in a model to change over time. However, application of the RCRP has largely been limited due to the non-conjugacy between the cluster evolutionary priors and the Multinomial likelihood. This non-conjugacy makes inference di cult and restricts the scalability of models which use the RCRP, leading to the RCRP being applied only in simple problems, such as those that can be approximated by a single Gaussian emission. In this paper, we provide a novel solution for the non-conjugacy issues for the RCRP and an example of how to leverage our solution for one speci c problem - the social event discovery problem. By utilizing Sequential Monte Carlo methods in inference, our approach can be massively paralleled and is highly scalable, to the extent it can work on tens of millions of documents. We are able to generate high quality topical and location distributions of the clusters that can be directly interpreted as real social events, and our experimental results suggest that the approaches proposed achieve much better predictive performance than techniques reported in prior work. We also demonstrate how the techniques we develop can be used in a much more general ways toward similar problems.