iNLTK: Natural Language Toolkit for Indic Languages

TL;DR

iNLTK is an open-source NLP toolkit for 13 Indic languages that provides pre-trained models and supports various NLP tasks, significantly improving performance and data efficiency in text classification.

Contribution

This work introduces iNLTK, a comprehensive NLP library for Indic languages with pre-trained models and tools, enhancing performance and reducing data requirements.

Findings

Outperforms previous results on text classification datasets.

Achieves over 95% of best performance with less than 10% of training data.

Widely adopted with 40,000+ downloads and active community.

Abstract

We present iNLTK, an open-source NLP library consisting of pre-trained language models and out-of-the-box support for Data Augmentation, Textual Similarity, Sentence Embeddings, Word Embeddings, Tokenization and Text Generation in 13 Indic Languages. By using pre-trained models from iNLTK for text classification on publicly available datasets, we significantly outperform previously reported results. On these datasets, we also show that by using pre-trained models and data augmentation from iNLTK, we can achieve more than 95% of the previous best performance by using less than 10% of the training data. iNLTK is already being widely used by the community and has 40,000+ downloads, 600+ stars and 100+ forks on GitHub. The library is available at https://github.com/goru001/inltk.