Runtime-optimized Multi-way Stream Join Operator for Large-scale Streaming data

TL;DR

This paper presents a runtime-optimized multi-way stream join operator that adaptively adjusts probe order using statistical predictions, significantly improving processing efficiency for large-scale real-time data streams.

Contribution

It introduces an adaptive, runtime-optimized multi-way stream join operator with a novel cost-based probe order refinement method for better real-time performance.

Findings

Significantly outperforms existing methods in processing efficiency.

Uses quadratic exponential smoothing for accurate data stream prediction.

Employs a cost model-based algorithm for adaptive probe order optimization.

Abstract

Streaming computing enables the real-time processing of large volumes of data and offers significant advantages for various applications, including real-time recommendations, anomaly detection, and monitoring. The multi-way stream join operator facilitates the integration of multiple data streams into a single operator, allowing for a more comprehensive understanding by consolidating information from diverse sources. Although this operator is valuable in stream processing systems, its current probe order is determined prior to execution, making it challenging to adapt to real-time and unpredictable data streams, which can potentially diminish its operational efficiency. In this paper, we introduce a runtime-optimized multi-way stream join operator that incorporates various adaptive strategies to enhance the probe order during the joining of multi-way data streams. The operator's runtime operation is divided into cycles, during which relevant statistical information from the data streams is collected and updated. Historical statistical data is then utilized to predict the characteristics of the data streams in the current cycle using a quadratic exponential smoothing prediction method. An adaptive optimization algorithm based on a cost model, namely dpPick, is subsequently designed to refine the probe order, enabling better adaptation to real-time, unknown data streams and improving the operator's processing efficiency. Experiments conducted on the TPC-DS dataset demonstrate that the proposed multi-way stream join method significantly outperforms the comparative method in terms of processing efficiency.