Shallow Parsing Pipeline for Hindi-English Code-Mixed Social Media Text

TL;DR

This paper presents the first shallow parsing pipeline for Hindi-English code-mixed social media text, including annotation, language identification, normalization, POS tagging, and shallow parsing, to improve text analysis.

Contribution

It introduces a comprehensive shallow parsing pipeline specifically designed for Hindi-English code-mixed social media data, which was not previously available.

Findings

Pipeline is publicly accessible for research use

Achieved initial success in shallow parsing of CSMT

Provides foundational tools for better social media text analysis

Abstract

In this study, the problem of shallow parsing of Hindi-English code-mixed social media text (CSMT) has been addressed. We have annotated the data, developed a language identifier, a normalizer, a part-of-speech tagger and a shallow parser. To the best of our knowledge, we are the first to attempt shallow parsing on CSMT. The pipeline developed has been made available to the research community with the goal of enabling better text analysis of Hindi English CSMT. The pipeline is accessible at http://bit.ly/csmt-parser-api .