Adaptive Normalization in Streaming Data

TL;DR

This paper introduces a distributed, adaptive normalization method for streaming Big Data that dynamically adjusts to data changes using sliding windows, improving normalization accuracy and efficiency in real-time processing frameworks.

Contribution

It presents a novel distributed adaptive normalization technique for Big Data streams that operates without being tailored to specific tasks and offers adjustable tradeoffs between speed and accuracy.

Findings

Achieved 89% improvement over baseline in normalization accuracy.

Normalized 160,000 data instances with low RMS error of 0.0041.

Implemented on Apache Storm for real-time, scalable data processing.

Abstract

In todays digital era, data are everywhere from Internet of Things to health care or financial applications. This leads to potentially unbounded ever-growing Big data streams and it needs to be utilized effectively. Data normalization is an important preprocessing technique for data analytics. It helps prevent mismodeling and reduce the complexity inherent in the data especially for data integrated from multiple sources and contexts. Normalization of Big Data stream is challenging because of evolving inconsistencies, time and memory constraints, and non-availability of whole data beforehand. This paper proposes a distributed approach to adaptive normalization for Big data stream. Using sliding windows of fixed size, it provides a simple mechanism to adapt the statistics for normalizing changing data in each window. Implemented on Apache Storm, a distributed real-time stream data framework, our approach exploits distributed data processing for efficient normalization. Unlike other existing adaptive approaches that normalize data for a specific use (e.g., classification), ours does not. Moreover, our adaptive mechanism allows flexible controls, via user-specified thresholds, for normalization tradeoffs between time and precision. The paper illustrates our proposed approach along with a few other techniques and experiments on both synthesized and real-world data. The normalized data obtained from our proposed approach, on 160,000 instances of data stream, improves over the baseline by 89% with 0.0041 root-mean-square error compared with the actual data.