Fast $q$-gram Mining on SLP Compressed Strings

TL;DR

This paper introduces efficient algorithms for computing $q$-gram frequencies directly on strings compressed as SLPs, significantly improving performance for small $q$ in practical data mining and classification tasks.

Contribution

The authors develop the first $O(qn)$ time and space algorithms for $q$-gram frequency computation on SLP-compressed strings, demonstrating practical efficiency.

Findings

Algorithms outperform uncompressed text methods for small $q$

Experiments confirm practical speedups on real data

Applications in data mining and classification are feasible

Abstract

We present simple and efficient algorithms for calculating $q$-gram frequencies on strings represented in compressed form, namely, as a straight line program (SLP). Given an SLP of size $n$ that represents string $T$, we present an $O(qn)$ time and space algorithm that computes the occurrence frequencies of $q$-grams in $T$. Computational experiments show that our algorithm and its variation are practical for small $q$, actually running faster on various real string data, compared to algorithms that work on the uncompressed text. We also discuss applications in data mining and classification of string data, for which our algorithms can be useful.