OmniSketch: Efficient Multi-Dimensional High-Velocity Stream Analytics with Arbitrary Predicates

TL;DR

OmniSketch is a novel data sketching technique that efficiently supports complex multi-attribute aggregate queries with filters on high-velocity data streams, providing probabilistic guarantees and outperforming existing methods.

Contribution

It introduces the first scalable sketch for multi-attribute streams that supports arbitrary filters and aggregates with probabilistic guarantees and efficient updates.

Findings

Outperforms state-of-the-art sketches in accuracy and efficiency.

Supports complex ad-hoc queries with small memory footprint.

Provides probabilistic guarantees with logarithmic update and query complexity.

Abstract

A key need in different disciplines is to perform analytics over fast-paced data streams, similar in nature to the traditional OLAP analytics in relational databases i.e., with filters and aggregates. Storing unbounded streams, however, is not a realistic, or desired approach due to the high storage requirements, and the delays introduced when storing massive data. Accordingly, many synopses/sketches have been proposed that can summarize the stream in small memory (usually sufficiently small to be stored in RAM), such that aggregate queries can be efficiently approximated, without storing the full stream. However, past synopses predominantly focus on summarizing single-attribute streams, and cannot handle filters and constraints on arbitrary subsets of multiple attributes efficiently. In this work, we propose OmniSketch, the first sketch that scales to fast-paced and complex data streams (with many attributes), and supports aggregates with filters on multiple attributes, dynamically chosen at query time. The sketch offers probabilistic guarantees, a favorable space-accuracy tradeoff, and a worst-case logarithmic complexity for updating and for query execution. We demonstrate experimentally with both real and synthetic data that the sketch outperforms the state-of-the-art, and that it can approximate complex ad-hoc queries within the configured accuracy guarantees, with small memory requirements.