Sampling-Based Estimation of Jaccard Containment and Similarity

TL;DR

This paper proposes a sampling-based method for estimating set containment and similarity, providing accurate predictions with small samples without full set sketches, useful for large-scale data systems.

Contribution

Introduces a binomial model for estimating set overlap from samples, outperforming previous methods and analyzing statistical properties for practical application.

Findings

The binomial model accurately predicts set overlap from small samples.

The method provides reliable estimates with theoretical error bounds.

Guidelines for applying the approach in large-scale data systems.

Abstract

This paper addresses the problem of estimating the containment and similarity between two sets using only random samples from each set, without relying on sketches of full sets. The study introduces a binomial model for predicting the overlap between samples, demonstrating that it is both accurate and practical when sample sizes are small compared to the original sets. The paper compares this model to previous approaches and shows that it provides better estimates under the considered conditions. It also analyzes the statistical properties of the estimator, including error bounds and sample size requirements needed to achieve a desired level of accuracy and confidence. The framework is extended to estimate set similarity, and the paper provides guidance for applying these methods in large scale data systems where only partial or sampled data is available.