Improved Fast Similarity Search in Dictionaries

TL;DR

This paper introduces an optimized algorithm and data structures for fast approximate dictionary matching, enabling near-instantaneous retrieval of similar words within large datasets.

Contribution

The authors develop a novel, memory-efficient indexing method that significantly accelerates approximate string matching in large dictionaries.

Findings

Supports fault-tolerant queries with high speed

Reduces memory consumption and preprocessing time

Achieves microsecond query times on large datasets

Abstract

We engineer an algorithm to solve the approximate dictionary matching problem. Given a list of words $\mathcal{W}$, maximum distance $d$ fixed at preprocessing time and a query word $q$, we would like to retrieve all words from $\mathcal{W}$ that can be transformed into $q$ with $d$ or less edit operations. We present data structures that support fault tolerant queries by generating an index. On top of that, we present a generalization of the method that eases memory consumption and preprocessing time significantly. At the same time, running times of queries are virtually unaffected. We are able to match in lists of hundreds of thousands of words and beyond within microseconds for reasonable distances.