Hierarchical Character-Word Models for Language Identification

TL;DR

This paper presents a hierarchical model that combines character and word-level representations to improve language identification in social media messages, effectively handling challenges like brevity and unconventional spelling.

Contribution

The paper introduces a novel hierarchical character-word model that enhances language identification accuracy and can detect code-switching in social media text.

Findings

Outperforms strong baseline models

Effective in identifying language in brief, informal texts

Capable of revealing code-switching instances

Abstract

Social media messages' brevity and unconventional spelling pose a challenge to language identification. We introduce a hierarchical model that learns character and contextualized word-level representations for language identification. Our method performs well against strong base- lines, and can also reveal code-switching.