Universal Dependency Parsing for Hindi-English Code-switching

TL;DR

This paper develops a neural dependency parser for Hindi-English code-switching Twitter data, creating a new treebank, and introduces normalization and back-transliteration models to improve parsing accuracy.

Contribution

It presents a novel Hindi-English code-switching treebank under Universal Dependencies and a neural stacking parser that leverages multilingual syntactic information.

Findings

Neural stacking parser outperforms augmented models by 1.5% LAS.

Decoding process improves LAS by 3.8% over baseline normalization.

Created a new code-switching treebank for Hindi-English Twitter data.

Abstract

Code-switching is a phenomenon of mixing grammatical structures of two or more languages under varied social constraints. The code-switching data differ so radically from the benchmark corpora used in NLP community that the application of standard technologies to these data degrades their performance sharply. Unlike standard corpora, these data often need to go through additional processes such as language identification, normalization and/or back-transliteration for their efficient processing. In this paper, we investigate these indispensable processes and other problems associated with syntactic parsing of code-switching data and propose methods to mitigate their effects. In particular, we study dependency parsing of code-switching data of Hindi and English multilingual speakers from Twitter. We present a treebank of Hindi-English code-switching tweets under Universal Dependencies scheme and propose a neural stacking model for parsing that efficiently leverages part-of-speech tag and syntactic tree annotations in the code-switching treebank and the preexisting Hindi and English treebanks. We also present normalization and back-transliteration models with a decoding process tailored for code-switching data. Results show that our neural stacking parser is 1.5% LAS points better than the augmented parsing model and our decoding process improves results by 3.8% LAS points over the first-best normalization and/or back-transliteration.