VNLP: Turkish NLP Package

TL;DR

VNLP is a comprehensive, open-source NLP toolkit tailored for Turkish, offering a wide range of tools, models, and pre-trained resources to facilitate various language processing tasks in a production environment.

Contribution

It introduces the first complete Turkish NLP package with a novel context-based token classification architecture and extensive pre-trained resources.

Findings

Includes state-of-the-art models for multiple NLP tasks

Provides pre-trained embeddings and tokenizers

Offers a user-friendly API and documentation

Abstract

In this work, we present VNLP: the first dedicated, complete, open-source, well-documented, lightweight, production-ready, state-of-the-art Natural Language Processing (NLP) package for the Turkish language. It contains a wide variety of tools, ranging from the simplest tasks, such as sentence splitting and text normalization, to the more advanced ones, such as text and token classification models. Its token classification models are based on "Context Model", a novel architecture that is both an encoder and an auto-regressive model. NLP tasks solved by VNLP models include but are not limited to Sentiment Analysis, Named Entity Recognition, Morphological Analysis \& Disambiguation and Part-of-Speech Tagging. Moreover, it comes with pre-trained word embeddings and corresponding SentencePiece Unigram tokenizers. VNLP has an open-source GitHub repository, ReadtheDocs documentation, PyPi package for convenient installation, Python and command-line API and a demo page to test all the functionality. Consequently, our main contribution is a complete, compact, easy-to-install and easy-to-use NLP package for Turkish.