Exploiting the English Vocabulary Profile for L2 word-level vocabulary assessment with LLMs

TL;DR

This paper presents a novel method combining large language models with the English Vocabulary Profile to assess second language vocabulary use at the word level, addressing contextual and semantic nuances for improved proficiency evaluation.

Contribution

It introduces a new approach that leverages LLMs and the EVP for fine-grained, context-aware vocabulary assessment in L2 writing, surpassing traditional PoS-based methods.

Findings

LLMs outperform PoS-based baselines in assigning proficiency levels.

Semantic information from LLMs enhances vocabulary assessment accuracy.

The approach reveals correlations between word-level and essay-level proficiency.

Abstract

Vocabulary use is a fundamental aspect of second language (L2) proficiency. To date, its assessment by automated systems has typically examined the context-independent, or part-of-speech (PoS) related use of words. This paper introduces a novel approach to enable fine-grained vocabulary evaluation exploiting the precise use of words within a sentence. The scheme combines large language models (LLMs) with the English Vocabulary Profile (EVP). The EVP is a standard lexical resource that enables in-context vocabulary use to be linked with proficiency level. We evaluate the ability of LLMs to assign proficiency levels to individual words as they appear in L2 learner writing, addressing key challenges such as polysemy, contextual variation, and multi-word expressions. We compare LLMs to a PoS-based baseline. LLMs appear to exploit additional semantic information that yields improved performance. We also explore correlations between word-level proficiency and essay-level proficiency. Finally, the approach is applied to examine the consistency of the EVP proficiency levels. Results show that LLMs are well-suited for the task of vocabulary assessment.