Leveraging Open-Source Large Language Models for Native Language Identification

TL;DR

This paper investigates the use of open-source large language models for native language identification, showing that fine-tuning enables them to match the performance of proprietary models, despite lower out-of-the-box accuracy.

Contribution

It demonstrates that open-source LLMs, when fine-tuned, can perform comparably to closed-source models in NLI tasks, addressing cost and transparency issues.

Findings

Open-source LLMs underperform out-of-the-box compared to closed-source models.

Fine-tuning improves open-source LLM performance to match commercial models.

Open-source models offer a viable, transparent alternative for NLI after fine-tuning.

Abstract

Native Language Identification (NLI) - the task of identifying the native language (L1) of a person based on their writing in the second language (L2) - has applications in forensics, marketing, and second language acquisition. Historically, conventional machine learning approaches that heavily rely on extensive feature engineering have outperformed transformer-based language models on this task. Recently, closed-source generative large language models (LLMs), e.g., GPT-4, have demonstrated remarkable performance on NLI in a zero-shot setting, including promising results in open-set classification. However, closed-source LLMs have many disadvantages, such as high costs and undisclosed nature of training data. This study explores the potential of using open-source LLMs for NLI. Our results indicate that open-source LLMs do not reach the accuracy levels of closed-source LLMs when used out-of-the-box. However, when fine-tuned on labeled training data, open-source LLMs can achieve performance comparable to that of commercial LLMs.