Real-time intelligent big data processing: technology, platform, and applications

TL;DR

This paper introduces Stream Cube, an innovative technology integrating batch and streaming data processing to enable real-time intelligent data analysis and decision-making, addressing the needs of modern big data applications.

Contribution

It proposes a novel incremental processing technology called Stream Cube and implements a comprehensive real-time data processing system combining big data, analysis, and machine learning.

Findings

Effective real-time processing of big and stream data demonstrated

System supports real-time acquisition, analysis, and decision-making

Applicable to societal and economic challenges

Abstract

Human beings keep exploring the physical space using information means. Only recently, with the rapid development of information technologies and the increasing accumulation of data, human beings can learn more about the unknown world with data-driven methods. Given data timeliness, there is a growing awareness of the importance of real-time data. There are two categories of technologies accounting for data processing: batching big data and streaming processing, which have not been integrated well. Thus, we propose an innovative incremental processing technology named after Stream Cube to process both big data and stream data. Also, we implement a real-time intelligent data processing system, which is based on real-time acquisition, real-time processing, real-time analysis, and real-time decision-making. The real-time intelligent data processing technology system is equipped with a batching big data platform, data analysis tools, and machine learning models. Based on our applications and analysis, the real-time intelligent data processing system is a crucial solution to the problems of the national society and economy.