Scaling Native Language Identification with Transformer Adapters

TL;DR

This paper explores the use of transformer adapters to enhance native language identification systems, focusing on scalability, efficiency, and practical deployment compared to traditional methods and transformer decoders.

Contribution

It introduces transformer adapters for NLI to overcome memory constraints and accelerate training and inference, enabling scalable production-ready systems.

Findings

Transformer adapters improve training speed and memory efficiency.

The approach scales NLI applications for real-world deployment.

Transformer decoders outperform traditional features in NLI tasks.

Abstract

Native language identification (NLI) is the task of automatically identifying the native language (L1) of an individual based on their language production in a learned language. It is useful for a variety of purposes including marketing, security and educational applications. NLI is usually framed as a multi-label classification task, where numerous designed features are combined to achieve state-of-the-art results. Recently deep generative approach based on transformer decoders (GPT-2) outperformed its counterparts and achieved the best results on the NLI benchmark datasets. We investigate this approach to determine the practical implications compared to traditional state-of-the-art NLI systems. We introduce transformer adapters to address memory limitations and improve training/inference speed to scale NLI applications for production.