Efficient Join Processing Over Incomplete Data Streams (Technical Report)

TL;DR

This paper introduces a novel approach for join processing over incomplete data streams, combining data imputation and query execution to improve efficiency and accuracy in real-time applications.

Contribution

It proposes a cost-model-based imputation method using differential dependency, along with pruning strategies and efficient algorithms for join over incomplete streams.

Findings

The approach achieves high confidence in join results.

Experimental results show improved efficiency over existing methods.

Effective handling of missing data in high-speed streams.

Abstract

For decades, the join operator over fast data streams has always drawn much attention from the database community, due to its wide spectrum of real-world applications, such as online clustering, intrusion detection, sensor data monitoring, and so on. Existing works usually assume that the underlying streams to be joined are complete (without any missing values). However, this assumption may not always hold, since objects from streams may contain some missing attributes, due to various reasons such as packet losses, network congestion/failure, and so on. In this paper, we formalize an important problem, namely join over incomplete data streams (Join-iDS), which retrieves joining object pairs from incomplete data streams with high confidences. We tackle the Join-iDS problem in the style of "data imputation and query processing at the same time". To enable this style, we design an effective and efficient cost-model-based imputation method via deferential dependency (DD), devise effective pruning strategies to reduce the Join-iDS search space, and propose efficient algorithms via our proposed cost-model-based data synopsis/indexes. Extensive experiments have been conducted to verify the efficiency and effectiveness of our proposed Join-iDS approach on both real and synthetic data sets.