FluxSieve: Unifying Streaming and Analytical Data Planes for Scalable Cloud Observability

TL;DR

FluxSieve introduces a unified data processing architecture that embeds in-stream filtering during data ingestion, significantly improving query performance in large-scale observability platforms with minimal overhead.

Contribution

It presents a novel architecture unifying streaming and analytical data planes through in-stream filtering and records enrichment, enabling scalable, high-performance query processing.

Findings

Up to orders-of-magnitude query performance improvements

Minimal additional storage and low computational overhead

Effective integration with Apache Pinot and DuckDB

Abstract

Despite many advances in query optimization, indexing techniques, and data storage, modern data platforms still face difficulties in delivering robust query performance under high concurrency and computationally intensive queries. This challenge is particularly pronounced in large-scale observability platforms handling high-volume, high-velocity data records. For instance, recurrent, expensive filtering queries at query time impose substantial computational and storage overheads in the analytical data plane. In this paper, we propose FluxSieve, a unified architecture that reconciles traditional pull-based query processing with push-based stream processing by embedding a lightweight in-stream precomputation and filtering layer directly into the data ingestion path. This avoids the complexity and operational burden of running queries in dedicated stream processing frameworks. Concretely, this work (i) introduces a foundational architecture that unifies streaming and analytical data planes via in-stream filtering and records enrichment, (ii) designs a scalable multi-pattern matching mechanism that supports concurrent evaluation and on-the-fly updates of filtering rules with minimal per-record overhead, (iii) demonstrates how to integrate this ingestion-time processing with two open-source analytical systems -- Apache Pinot as a Real-Time Online Analytical Processing (RTOLAP) engine and DuckDB as an embedded analytical database, and (iv) performs comprehensive experimental evaluation of our approach. Our evaluation across different systems, query types, and performance metrics shows up to orders-of-magnitude improvements in query performance at the cost of negligible additional storage and very low computational overhead.