Non-native Speaker Verification for Spoken Language Assessment

TL;DR

This paper explores the application of deep-learning-based speaker verification systems to detect impersonation malpractice in spoken language assessments, focusing on non-native English speakers and analyzing factors affecting performance.

Contribution

It introduces the adaptation of speaker verification models to non-native English data in language assessment, highlighting challenges with inter-L1 impostors and providing empirical evaluation results.

Findings

Best performance achieved by adapting models to non-native data

Inter-L1 impostors pose greater challenges for verification systems

Large-scale experiments demonstrate effectiveness with millions of trials

Abstract

Automatic spoken language assessment systems are becoming more popular in order to handle increasing interests in second language learning. One challenge for these systems is to detect malpractice. Malpractice can take a range of forms, this paper focuses on detecting when a candidate attempts to impersonate another in a speaking test. This form of malpractice is closely related to speaker verification, but applied in the specific domain of spoken language assessment. Advanced speaker verification systems, which leverage deep-learning approaches to extract speaker representations, have been successfully applied to a range of native speaker verification tasks. These systems are explored for non-native spoken English data in this paper. The data used for speaker enrolment and verification is mainly taken from the BULATS test, which assesses English language skills for business. Performance of systems trained on relatively limited amounts of BULATS data, and standard large speaker verification corpora, is compared. Experimental results on large-scale test sets with millions of trials show that the best performance is achieved by adapting the imported model to non-native data. Breakdown of impostor trials across different first languages (L1s) and grades is analysed, which shows that inter-L1 impostors are more challenging for speaker verification systems.