Analytical confidence intervals for the number of different objects in data streams

TL;DR

This paper introduces a rigorous mathematical approach to derive confidence intervals for the number of distinct objects in data streams using Flajolet-Martin algorithms, connecting statistical analysis with special functions and extreme value theory.

Contribution

It provides the first analytical confidence intervals for F0 in data streams, based on Chernoff bounds, with deep connections to special functions and extreme value theory.

Findings

Analytical confidence intervals for F0 derived using Chernoff bounds.

Connections established between statistical analysis, special functions, and extreme value theory.

Validation through real data stream tests and Monte Carlo simulations.

Abstract

This paper develops a new mathematical-statistical approach to analyze a class of Flajolet-Martin algorithms (FMa), and provides analytical confidence intervals for the number F0 of distinct elements in a stream, based on Chernoff bounds. The class of FMa has reached a significant popularity in bigdata stream learning, and the attention of the literature has mainly been based on algorithmic aspects, basically complexity optimality, while the statistical analysis of these class of algorithms has been often faced heuristically. The analysis provided here shows deep connections with mathematical special functions and with extreme value theory. The latter connection may help in explaining heuristic considerations, while the first opens many numerical issues, faced at the end of the present paper. Finally, the algorithms are tested on an anonymized real data stream and MonteCarlo simulations are provided to support our analytical choice in this context.