Speeding-up $q$-gram mining on grammar-based compressed texts

TL;DR

This paper introduces an efficient algorithm for computing $q$-gram frequencies directly on grammar-compressed texts, significantly speeding up the process by reducing redundancy and leveraging a trie-like structure.

Contribution

The authors develop a novel linear-time algorithm that improves previous methods by exploiting redundancy in grammar-based compressed texts for $q$-gram frequency computation.

Findings

Algorithm runs in $O( ext{min}igrace{|T|- ext{dup}(q, ext{SLP}), qnigrace})$ time.

Reduces complexity compared to previous $O(qn)$ algorithms when $q = ext{Omega}(|T|/n)$.

Effectively leverages redundancy in SLPs to optimize $q$-gram frequency calculations.

Abstract

We present an efficient algorithm for calculating $q$-gram frequencies on strings represented in compressed form, namely, as a straight line program (SLP). Given an SLP $\mathcal{T}$ of size $n$ that represents string $T$, the algorithm computes the occurrence frequencies of all $q$-grams in $T$, by reducing the problem to the weighted $q$-gram frequencies problem on a trie-like structure of size $m = |T|-\mathit{dup}(q,\mathcal{T})$, where $\mathit{dup}(q,\mathcal{T})$ is a quantity that represents the amount of redundancy that the SLP captures with respect to $q$-grams. The reduced problem can be solved in linear time. Since $m = O(qn)$, the running time of our algorithm is $O(\min\{|T|-\mathit{dup}(q,\mathcal{T}),qn\})$, improving our previous $O(qn)$ algorithm when $q = \Omega(|T|/n)$.