Fast Counting in Machine Learning Applications

TL;DR

This paper introduces scalable, memory-efficient methods for counting queries in machine learning, outperforming traditional data structures and enabling large-scale data processing.

Contribution

It presents a novel streaming aggregation approach for counting queries that improves efficiency and scalability in machine learning tasks.

Findings

Outperforms ADtrees and hash tables in speed and memory usage.

Effective in Bayesian network learning and association rule mining.

Demonstrates scalability on large datasets.

Abstract

We propose scalable methods to execute counting queries in machine learning applications. To achieve memory and computational efficiency, we abstract counting queries and their context such that the counts can be aggregated as a stream. We demonstrate performance and scalability of the resulting approach on random queries, and through extensive experimentation using Bayesian networks learning and association rule mining. Our methods significantly outperform commonly used ADtrees and hash tables, and are practical alternatives for processing large-scale data.