Optimizing Exact String Matching via Statistical Anchoring

TL;DR

This paper introduces a statistical anchoring technique to optimize string matching by preprocessing patterns to identify a low-frequency character, enabling faster verification and improved efficiency in natural language processing tasks.

Contribution

It presents a novel enhancement to the Boyer-Moore-Horspool algorithm using linguistic statistics to identify an anchor character for faster pattern matching.

Findings

Significant reduction in comparison counts during search

Improved matching speed without added algorithm complexity

Effective for natural language text processing

Abstract

In this work, we propose an enhancement to the Boyer-Moore-Horspool algorithm tailored for natural language text. The approach involves preprocessing the search pattern to identify its statistically least frequent character, referred to as the "anchor." During the search, verification is first performed at this high-entropy position, allowing the algorithm to quickly discard non-matching windows. This fail-fast strategy reduces unnecessary comparisons, improving overall efficiency. Our implementation shows that incorporating basic linguistic statistics into classical pattern-matching techniques can boost performance without increasing complexity to the shift heuristics.