Can string kernels pass the test of time in Native Language Identification?

TL;DR

This paper demonstrates that simple string kernel methods, combined with multiple kernel learning, can achieve state-of-the-art results in Native Language Identification, outperforming recent NLP advances in the 2017 shared task.

Contribution

The study shows that a shallow, string kernel-based approach with minor improvements remains competitive and effective for NLI, even compared to modern NLP techniques.

Findings

Achieved top macro F1 scores in all three NLI tracks.

Outperformed other methods in speech and fusion tracks.

Validated the effectiveness of string kernels in NLI tasks.

Abstract

We describe a machine learning approach for the 2017 shared task on Native Language Identification (NLI). The proposed approach combines several kernels using multiple kernel learning. While most of our kernels are based on character p-grams (also known as n-grams) extracted from essays or speech transcripts, we also use a kernel based on i-vectors, a low-dimensional representation of audio recordings, provided by the shared task organizers. For the learning stage, we choose Kernel Discriminant Analysis (KDA) over Kernel Ridge Regression (KRR), because the former classifier obtains better results than the latter one on the development set. In our previous work, we have used a similar machine learning approach to achieve state-of-the-art NLI results. The goal of this paper is to demonstrate that our shallow and simple approach based on string kernels (with minor improvements) can pass the test of time and reach state-of-the-art performance in the 2017 NLI shared task, despite the recent advances in natural language processing. We participated in all three tracks, in which the competitors were allowed to use only the essays (essay track), only the speech transcripts (speech track), or both (fusion track). Using only the data provided by the organizers for training our models, we have reached a macro F1 score of 86.95% in the closed essay track, a macro F1 score of 87.55% in the closed speech track, and a macro F1 score of 93.19% in the closed fusion track. With these scores, our team (UnibucKernel) ranked in the first group of teams in all three tracks, while attaining the best scores in the speech and the fusion tracks.