SoftMatcha 2: A Fast and Soft Pattern Matcher for Trillion-Scale Corpora

TL;DR

SoftMatcha 2 introduces a rapid, flexible pattern matching algorithm capable of searching trillion-scale corpora in under 0.3 seconds, effectively handling semantic variations and outperforming existing methods in speed and utility.

Contribution

The paper presents a novel string matching algorithm based on suffix arrays that scales efficiently with large corpora and incorporates semantic relaxation with pruning to control search space growth.

Findings

Achieves search over 1.4 trillion tokens in under 0.3 seconds.

Significantly outperforms existing search methods like infini-gram and SoftMatcha.

Effectively identifies training data contamination in large corpora.

Abstract

We present an ultra-fast and flexible search algorithm that enables search over trillion-scale natural language corpora in under 0.3 seconds while handling semantic variations (substitution, insertion, and deletion). Our approach employs string matching based on suffix arrays that scales well with corpus size. To mitigate the combinatorial explosion induced by the semantic relaxation of queries, our method is built on two key algorithmic ideas: fast exact lookup enabled by a disk-aware design, and dynamic corpus-aware pruning. We theoretically show that the proposed method suppresses exponential growth in the search space with respect to query length by leveraging statistical properties of natural language. In experiments on FineWeb-Edu (Lozhkov et al., 2024) (1.4T tokens), we show that our method achieves significantly lower search latency than existing methods: infini-gram (Liu et al., 2024), infini-gram mini (Xu et al., 2025), and SoftMatcha (Deguchi et al., 2025). As a practical application, we demonstrate that our method identifies benchmark contamination in training corpora, unidentified by existing approaches. We also provide an online demo of fast, soft search across corpora in seven languages.