Open-Set Language Identification

TL;DR

This paper introduces a novel open-set language identification method using one-class classification with hashing-based feature vectors, achieving high accuracy across multiple languages with different writing systems.

Contribution

It proposes a new hashing-based feature vectorization approach and demonstrates its effectiveness for open-set language identification with one-class classifiers.

Findings

Achieved an average F-score of 0.99 across 10 languages.

Identified shortcomings of traditional feature extraction methods.

Validated the approach on diverse writing systems.

Abstract

We present the first open-set language identification experiments using one-class classification. We first highlight the shortcomings of traditional feature extraction methods and propose a hashing-based feature vectorization approach as a solution. Using a dataset of 10 languages from different writing systems, we train a One- Class Support Vector Machine using only a monolingual corpus for each language. Each model is evaluated against a test set of data from all 10 languages and we achieve an average F-score of 0.99, highlighting the effectiveness of this approach for open-set language identification.