Document Counting in Practice
Travis Gagie, Aleksi Hartikainen, Juha K\"arkk\"ainen, Gonzalo, Navarro, Simon J. Puglisi, Jouni Sir\'en

TL;DR
This paper evaluates practical implementations of string counting methods in data collections, comparing their efficiency and uncovering compressibility properties that significantly reduce data structure sizes.
Contribution
It implements existing theoretical solutions, develops new variants, and experimentally compares them, revealing compressibility properties that optimize data structure sizes.
Findings
Identifies the most effective solutions for different datasets
Discovers unexpected compressibility properties of data structures
Achieves size reductions by factors of 5 to 400
Abstract
We address the problem of counting the number of strings in a collection where a given pattern appears, which has applications in information retrieval and data mining. Existing solutions are in a theoretical stage. We implement these solutions and develop some new variants, comparing them experimentally on various datasets. Our results not only show which are the best options for each situation and help discard practically unappealing solutions, but also uncover some unexpected compressibility properties of the best data structures. By taking advantage of these properties, we can reduce the size of the structures by a factor of 5--400, depending on the dataset.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAlgorithms and Data Compression · Data Management and Algorithms · Data Mining Algorithms and Applications
