Mining Large-Scale Low-Resource Pronunciation Data From Wikipedia

TL;DR

This paper presents a system that extracts pronunciation data, including phoneme inventories and G2P mappings, from Wikipedia's loosely structured tables for 819 languages, aiding speech technology development in low-resource languages.

Contribution

The authors developed a method to mine and structure pronunciation data from Wikipedia for 819 languages, including 63 low-resource languages with G2P mappings, and made the dataset publicly available.

Findings

Extracted data for 819 languages from Wikipedia

Provided G2P mappings for 63 low-resource languages

Made the dataset publicly accessible for further research

Abstract

Pronunciation modeling is a key task for building speech technology in new languages, and while solid grapheme-to-phoneme (G2P) mapping systems exist, language coverage can stand to be improved. The information needed to build G2P models for many more languages can easily be found on Wikipedia, but unfortunately, it is stored in disparate formats. We report on a system we built to mine a pronunciation data set in 819 languages from loosely structured tables within Wikipedia. The data includes phoneme inventories, and for 63 low-resource languages, also includes the grapheme-to-phoneme (G2P) mapping. 54 of these languages do not have easily findable G2P mappings online otherwise. We turned the information from Wikipedia into a structured, machine-readable TSV format, and make the resulting data set publicly available so it can be improved further and used in a variety of applications involving low-resource languages.