Sorting the Babble in Babel: Assessing the Performance of Language Identification Algorithms on the OpenAlex Database

TL;DR

This study evaluates various language identification algorithms on the OpenAlex database, analyzing their accuracy and speed to optimize linguistic indexing for large-scale bibliographic data.

Contribution

It provides a comparative analysis of Python-based language ID algorithms on OpenAlex metadata, highlighting optimal choices based on precision, recall, and processing speed.

Findings

FastText on Titles corpus outperforms others when recall or speed is prioritized.

Performance varies significantly depending on the importance assigned to precision or recall.

Results support the use of OpenAlex for multilingual and comprehensive bibliometric analysis.

Abstract

This project aims to optimize the linguistic indexing of the OpenAlex database by comparing the performance of various Python-based language identification procedures on different metadata corpora extracted from a manually-annotated article sample \footnote{OpenAlex used the results presented in this article to inform the language metadata overhaul carried out as part of its recent Walden system launch. The precision and recall performance of each algorithm, corpus, and language is first analyzed, followed by an assessment of processing speeds recorded for each algorithm and corpus type. These different performance measures are then simulated at the database level using probabilistic confusion matrices for each algorithm, corpus, and language, as well as a probabilistic modeling of relative article language frequencies for the whole OpenAlex database. Results show that procedure performance strongly depends on the importance given to each of the measures implemented: for contexts where precision is preferred, using the LangID algorithm on the greedy corpus gives the best results; however, for all cases where recall is considered at least slightly more important than precision or as soon as processing times are given any kind of consideration, the procedure that consists in the application of the FastText algorithm on the Titles corpus outperforms all other alternatives. Given the lack of truly multilingual large-scale bibliographic databases, it is hoped that these results help confirm and foster the unparalleled potential of the OpenAlex database for cross-linguistic and comprehensive measurement and evaluation.