AMIC: An Adaptive Information Theoretic Method to Identify Multi-Scale Temporal Correlations in Big Time Series Data -- Accepted Version

TL;DR

AMIC is a scalable, adaptive method leveraging mutual information to identify and prioritize multi-scale temporal correlations in large time series data, facilitating insights from Big Data.

Contribution

The paper introduces AMIC, a novel adaptive mutual information-based method for scalable, multi-scale correlation detection in big time series data, implemented on Apache Spark.

Findings

Effective detection of multi-scale correlations demonstrated.

Scalable performance validated on synthetic and real data.

Prioritized correlation ordering enhances analysis efficiency.

Abstract

Recent development in computing, sensing and crowd-sourced data have resulted in an explosion in the availability of quantitative information. The possibilities of analyzing this so-called Big Data to inform research and the decision-making process are virtually endless. In general, analyses have to be done across multiple data sets in order to bring out the most value of Big Data. A first important step is to identify temporal correlations between data sets. Given the characteristics of Big Data in terms of volume and velocity, techniques that identify correlations not only need to be fast and scalable, but also need to help users in ordering the correlations across temporal scales so that they can focus on important relationships. In this paper, we present AMIC (Adaptive Mutual Information-based Correlation), a method based on mutual information to identify correlations at multiple temporal scales in large time series. Discovered correlations are suggested to users in an order based on the strength of the relationships. Our method supports an adaptive streaming technique that minimizes duplicated computation and is implemented on top of Apache Spark for scalability. We also provide a comprehensive evaluation on the effectiveness and the scalability of AMIC using both synthetic and real-world data sets.