Measures of lexical distance between languages

TL;DR

This paper compares two methods of measuring lexical distance between languages, one based on normalized Levenshtein distance and a refined version, to determine which better resolves language relationships.

Contribution

It evaluates and compares the effectiveness of two lexical distance measures, including a recent refinement, for analyzing language relationships.

Findings

The refined normalization provides more accurate language relationship insights.

The original Levenshtein-based method is simpler and effective.

Comparison shows which measure better captures linguistic similarities.

Abstract

The idea of measuring distance between languages seems to have its roots in the work of the French explorer Dumont D'Urville \cite{Urv}. He collected comparative words lists of various languages during his voyages aboard the Astrolabe from 1826 to 1829 and, in his work about the geographical division of the Pacific, he proposed a method to measure the degree of relation among languages. The method used by modern glottochronology, developed by Morris Swadesh in the 1950s, measures distances from the percentage of shared cognates, which are words with a common historical origin. Recently, we proposed a new automated method which uses normalized Levenshtein distance among words with the same meaning and averages on the words contained in a list. Recently another group of scholars \cite{Bak, Hol} proposed a refined of our definition including a second normalization. In this paper we compare the information content of our definition with the refined version in order to decide which of the two can be applied with greater success to resolve relationships among languages.