Multilingual Open Text Release 1: Public Domain News in 44 Languages
Chester Palen-Michel, June Kim, Constantine Lignos

TL;DR
This paper introduces MOT, a large multilingual news corpus in 44 languages, providing valuable resources for NLP tasks, especially for low-resource languages, with open licensing and ongoing updates.
Contribution
The creation and release of a comprehensive, multilingual news corpus in 44 languages, including data collection, processing, licensing, and open-source tools.
Findings
Contains over 2.8 million news articles and 1 million snippets
Covers languages with limited NLP resources
Openly licensed and regularly updated
Abstract
We present Multilingual Open Text (MOT), a new multilingual corpus containing text in 44 languages, many of which have limited existing text resources for natural language processing. The first release of the corpus contains over 2.8 million news articles and an additional 1 million short snippets (photo captions, video descriptions, etc.) published between 2001--2022 and collected from Voice of America's news websites. We describe our process for collecting, filtering, and processing the data. The source material is in the public domain, our collection is licensed using a creative commons license (CC BY 4.0), and all software used to create the corpus is released under the MIT License. The corpus will be regularly updated as additional documents are published.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNatural Language Processing Techniques · Topic Modeling · Multimodal Machine Learning Applications
