Document Counting in Practice

TL;DR

This paper evaluates practical implementations of string counting methods in data collections, comparing their efficiency and uncovering compressibility properties that significantly reduce data structure sizes.

Contribution

It implements existing theoretical solutions, develops new variants, and experimentally compares them, revealing compressibility properties that optimize data structure sizes.

Findings

Identifies the most effective solutions for different datasets

Discovers unexpected compressibility properties of data structures

Achieves size reductions by factors of 5 to 400

Abstract

We address the problem of counting the number of strings in a collection where a given pattern appears, which has applications in information retrieval and data mining. Existing solutions are in a theoretical stage. We implement these solutions and develop some new variants, comparing them experimentally on various datasets. Our results not only show which are the best options for each situation and help discard practically unappealing solutions, but also uncover some unexpected compressibility properties of the best data structures. By taking advantage of these properties, we can reduce the size of the structures by a factor of 5--400, depending on the dataset.