MCRapper: Monte-Carlo Rademacher Averages for Poset Families and Approximate Pattern Mining

TL;DR

MCRapper introduces an efficient algorithm for computing Monte-Carlo Rademacher Averages in poset-structured families, enabling statistically sound pattern mining with improved accuracy and power over existing methods.

Contribution

The paper presents MCRapper, a novel algorithm that efficiently computes MCERA for poset families, unifying significance testing and high-expectation approximation in pattern mining.

Findings

MCRapper outperforms existing solutions in efficiency and accuracy.

TFP-R algorithm guarantees low false positive probability.

Experimental results show superior statistical power and performance.

Abstract

We present MCRapper, an algorithm for efficient computation of Monte-Carlo Empirical Rademacher Averages (MCERA) for families of functions exhibiting poset (e.g., lattice) structure, such as those that arise in many pattern mining tasks. The MCERA allows us to compute upper bounds to the maximum deviation of sample means from their expectations, thus it can be used to find both statistically-significant functions (i.e., patterns) when the available data is seen as a sample from an unknown distribution, and approximations of collections of high-expectation functions (e.g., frequent patterns) when the available data is a small sample from a large dataset. This feature is a strong improvement over previously proposed solutions that could only achieve one of the two. MCRapper uses upper bounds to the discrepancy of the functions to efficiently explore and prune the search space, a technique borrowed from pattern mining itself. To show the practical use of MCRapper, we employ it to develop an algorithm TFP-R for the task of True Frequent Pattern (TFP) mining. TFP-R gives guarantees on the probability of including any false positives (precision) and exhibits higher statistical power (recall) than existing methods offering the same guarantees. We evaluate MCRapper and TFP-R and show that they outperform the state-of-the-art for their respective tasks.