A Unified Approach for Multi-Scale Synchronous Correlation Search in Big Time Series -- Full Version

TL;DR

This paper introduces iSYCOS, a comprehensive framework for multi-scale correlation search in large time series data, capable of handling noise, scaling efficiently, and automatically configuring for diverse datasets.

Contribution

It presents a novel integrated framework combining top-down and bottom-up methods with noise detection and pruning, scalable on Spark clusters, for complex correlation discovery in big time series.

Findings

iSYCOS auto-configures to find complex multi-scale correlations.

Pruning and optimizations improve performance up to tenfold.

Distributed iSYCOS scales linearly on Spark clusters.

Abstract

The wide deployment of IoT sensors has enabled the collection of very big time series across different domains, from which advanced analytics can be performed to find unknown relationships, most importantly the correlations between them. However, current approaches for correlation search on time series are limited to only a single temporal scale and simple types of relations, and cannot handle noise effectively. This paper presents the integrated SYnchronous COrrelation Search (iSYCOS) framework to find multi-scale correlations in big time series. Specifically, iSYCOS integrates top-down and bottom-up approaches into a single auto-configured framework capable of efficiently extracting complex window-based correlations from big time series using mutual information (MI). Moreover, iSYCOS includes a novel MI-based theory to identify noise in the data, and is used to perform pruning to improve iSYCOS performance. Besides, we design a distributed version of iSYCOS that can scale out in a Spark cluster to handle big time series. Our extensive experimental evaluation on synthetic and real-world datasets shows that iSYCOS can auto-configure on a given dataset to find complex multi-scale correlations. The pruning and optimisations can improve iSYCOS performance up to an order of magnitude, and the distributed iSYCOS can scale out linearly on a computing cluster.