Detecting Structural Irregularity in Electronic Dictionaries Using Language Modeling

TL;DR

This paper presents a statistical language modeling approach to automatically detect structural irregularities in XML-based electronic dictionaries, aiding in efficient error identification and quality assurance.

Contribution

It introduces a novel method that learns XML node patterns to identify deviations in dictionary entries, improving error detection over manual review.

Findings

Effective identification of irregular entries using language models

Reduced manual effort in dictionary quality control

Demonstrated applicability across various dictionary structures

Abstract

Dictionaries are often developed using tools that save to Extensible Markup Language (XML)-based standards. These standards often allow high-level repeating elements to represent lexical entries, and utilize descendants of these repeating elements to represent the structure within each lexical entry, in the form of an XML tree. In many cases, dictionaries are published that have errors and inconsistencies that are expensive to find manually. This paper discusses a method for dictionary writers to quickly audit structural regularity across entries in a dictionary by using statistical language modeling. The approach learns the patterns of XML nodes that could occur within an XML tree, and then calculates the probability of each XML tree in the dictionary against these patterns to look for entries that diverge from the norm.