Scalable and Reliable Multi-Dimensional Aggregation of Sensor Data Streams

TL;DR

This paper introduces a scalable, reliable stream processing architecture for hierarchical multi-dimensional sensor data aggregation, supporting real-time reconfiguration and integration into existing big data systems, demonstrated through industrial application and experiments.

Contribution

It presents a novel architecture for hierarchical sensor data aggregation that supports multiple hierarchies, runtime reconfiguration, and integration with big data platforms.

Findings

Linear scalability with sensor count

Adequate fault tolerance and reliability

Successful industrial deployment

Abstract

Ever-increasing amounts of data and requirements to process them in real time lead to more and more analytics platforms and software systems being designed according to the concept of stream processing. A common area of application is the processing of continuous data streams from sensors, for example, IoT devices or performance monitoring tools. In addition to analyzing pure sensor data, analyses of data for groups of sensors often need to be performed as well. Therefore, data streams of the individual sensors have to be continuously aggregated to a data stream for a group. Motivated by a real-world application scenario, we propose that such a stream aggregation approach has to allow for aggregating sensors in hierarchical groups, support multiple such hierarchies in parallel, provide reconfiguration at runtime, and preserve the scalability and reliability qualities induced by applying stream processing techniques. We propose a stream processing architecture fulfilling these requirements, which can be integrated into existing big data architectures. We present a pilot implementation of such an extended architecture and show how it is used in industry. Furthermore, in experimental evaluations we show that our solution scales linearly with the amount of sensors and provides adequate reliability in the case of faults.