Intermediate N-Gramming: Deterministic and Fast N-Grams For Large N and Large Datasets

TL;DR

This paper introduces Intergrams, a deterministic and efficient multi-pass algorithm for extracting the most frequent n-grams from large datasets, achieving significant speedups over existing methods.

Contribution

The paper presents a novel multi-pass algorithm, Intergrams, optimized for hardware, that accurately and quickly recovers top-k n-grams for large n and datasets, with theoretical analysis.

Findings

Achieves up to 33x speedup over previous algorithms

Effective for large n and datasets due to multi-pass design

Theoretical analysis supports empirical results

Abstract

The number of n-gram features grows exponentially in n, making it computationally demanding to compute the most frequent n-grams even for n as small as 3. Motivated by our production machine learning system built on n-gram features, we ask: is it possible to accurately, deterministically, and quickly recover the top-k most frequent n-grams? We devise a multi-pass algorithm called Intergrams that constructs candidate n-grams from the preceding (n - 1)-grams. By designing this algorithm with hardware in mind, our approach yields more than an order of magnitude speedup (up to 33x!) over the next known fastest algorithm, even when similar optimizations are applied to the other algorithm. Using the empirical power-law distribution over n-grams, we also provide theory to inform the efficacy of our multi-pass approach. Our code is available at https://github.com/rcurtin/Intergrams.