Practical Algorithms for Finding Extremal Sets

TL;DR

This paper introduces two improvements to the extremal set identification algorithm AMS-Lex, using memoization and parallelization, resulting in significantly faster execution times on various datasets, including real-world and synthetic data.

Contribution

The paper presents novel memoization and parallelization techniques that substantially enhance the efficiency of the AMS-Lex algorithm for finding extremal sets.

Findings

Memoization improves execution time by over 400 times in some cases.

Multi-threaded implementation outperforms original AMS-Lex by 3 to 6 times.

Parallel CPU implementation matches GPU-based performance on synthetic datasets.

Abstract

The minimal sets within a collection of sets are defined as the ones which do not have a proper subset within the collection, and the maximal sets are the ones which do not have a proper superset within the collection. Identifying extremal sets is a fundamental problem with a wide-range of applications in SAT solvers, data-mining and social network analysis. In this paper, we present two novel improvements of the high-quality extremal set identification algorithm, \textit{AMS-Lex}, described by Bayardo and Panda. The first technique uses memoization to improve the execution time of the single-threaded variant of the AMS-Lex, whilst our second improvement uses parallel programming methods. In a subset of the presented experiments our memoized algorithm executes more than $400$ times faster than the highly efficient publicly available implementation of AMS-Lex. Moreover, we show that our modified algorithm's speedup is not bounded above by a constant and that it increases as the length of the common prefixes in successive input \textit{itemsets} increases. We provide experimental results using both real-world and synthetic data sets, and show our multi-threaded variant algorithm out-performing AMS-Lex by $3$ to $6$ times. We find that on synthetic input datasets when executed using $16$ CPU cores of a $32$-core machine, our multi-threaded program executes about as fast as the state of the art parallel GPU-based program using an NVIDIA GTX 580 graphics processing unit.