Imagined-Trailing-Whitespace-Agnostic Levenshtein Distance For Plaintext Table Detection

TL;DR

This paper introduces an efficient Levenshtein distance variant that accounts for human-like assumptions about trailing whitespace, improving string comparison in table detection tasks.

Contribution

It proposes a novel $O(n^2)$ algorithm for whitespace-agnostic Levenshtein distance that aligns with human intuition, unlike the standard approach.

Findings

The new algorithm is computationally efficient at $O(n^2)$.

It improves table detection accuracy by better modeling human string comparison.

The method is applicable to table alignment and related tasks.

Abstract

The standard algorithm for Levenshtein distance, treats trailing whitespace the same as any other letter or symbol. However, when humans compare 2 strings, we implicitly assume that both strings are padded by infinite trailing whitespace. This informs our expectations for what the costs for insertion, deletion and replacement, should be. This violation of our expectations results in non-intuitive edit distance values. To account for this specific human intuition, a naive approach which considers "all possible" substrings of trailing whitespace would yield an $O(n^3)$ algorithm. In this work, we provide an efficient $O(n^2)$ algorithm to compute the same. Keywords: Imagined Infinite Trailing Whitespace, Human Friendly, Intuitive Edit Distance, Table Detection, Table Alignment