Using Multiple Sources of Information for Constraint-Based Morphological Disambiguation

TL;DR

This thesis introduces a constraint-based morphological disambiguation system for complex languages like Turkish, combining handcrafted rules, learned constraints, and statistical data to achieve high accuracy and low ambiguity.

Contribution

It presents a novel multi-source approach that integrates rule-based, learned, and statistical information for morphological disambiguation in agglutinative languages.

Findings

Achieved 96-97% recall and 93-94% precision in disambiguation

Reduced unknown words to below 1% with secondary processing

Attained low ambiguity of about 1.02 to 1.03 parses per token

Abstract

This thesis presents a constraint-based morphological disambiguation approach that is applicable to languages with complex morphology--specifically agglutinative languages with productive inflectional and derivational morphological phenomena. For morphologically complex languages like Turkish, automatic morphological disambiguation involves selecting for each token morphological parse(s), with the right set of inflectional and derivational markers. Our system combines corpus independent hand-crafted constraint rules, constraint rules that are learned via unsupervised learning from a training corpus, and additional statistical information obtained from the corpus to be morphologically disambiguated. The hand-crafted rules are linguistically motivated and tuned to improve precision without sacrificing recall. In certain respects, our approach has been motivated by Brill's recent work, but with the observation that his transformational approach is not directly applicable to languages like Turkish. Our approach also uses a novel approach to unknown word processing by employing a secondary morphological processor which recovers any relevant inflectional and derivational information from a lexical item whose root is unknown. With this approach, well below 1% of the tokens remains as unknown in the texts we have experimented with. Our results indicate that by combining these hand-crafted, statistical and learned information sources, we can attain a recall of 96 to 97% with a corresponding precision of 93 to 94%, and ambiguity of 1.02 to 1.03 parses per token.