LanideNN: Multilingual Language Identification on Character Window
Tom Kocmi, Ond\v{r}ej Bojar

TL;DR
This paper introduces LanideNN, a neural network-based method for multilingual language identification that accurately detects language spans in texts with arbitrary language changes, performing well across various datasets and document lengths.
Contribution
The paper presents a novel Bidirectional RNN approach for language identification that handles arbitrary language switches and works effectively on short texts and diverse domains.
Findings
High accuracy in monolingual and multilingual tasks
Effective on short documents and across domains
Supports 131 languages across six datasets
Abstract
In language identification, a common first step in natural language processing, we want to automatically determine the language of some input text. Monolingual language identification assumes that the given document is written in one language. In multilingual language identification, the document is usually in two or three languages and we just want their names. We aim one step further and propose a method for textual language identification where languages can change arbitrarily and the goal is to identify the spans of each of the languages. Our method is based on Bidirectional Recurrent Neural Networks and it performs well in monolingual and multilingual language identification tasks on six datasets covering 131 languages. The method keeps the accuracy also for short documents and across domains, so it is ideal for off-the-shelf use without preparation of training data.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNatural Language Processing Techniques · Authorship Attribution and Profiling · Topic Modeling
