MuLVE, A Multi-Language Vocabulary Evaluation Data Set

TL;DR

MuLVE is a new multi-language dataset with real user answers for vocabulary evaluation, enabling more accurate feedback systems in language learning applications.

Contribution

It introduces MuLVE, a comprehensive dataset with real user data across multiple languages, and demonstrates its effectiveness for fine-tuning language models for vocabulary assessment.

Findings

Achieved over 95.5% accuracy and F2-score in vocabulary evaluation tasks.

Provided a publicly available dataset for multi-language vocabulary learning research.

Showed the effectiveness of fine-tuning BERT models on real user data.

Abstract

Vocabulary learning is vital to foreign language learning. Correct and adequate feedback is essential to successful and satisfying vocabulary training. However, many vocabulary and language evaluation systems perform on simple rules and do not account for real-life user learning data. This work introduces Multi-Language Vocabulary Evaluation Data Set (MuLVE), a data set consisting of vocabulary cards and real-life user answers, labeled indicating whether the user answer is correct or incorrect. The data source is user learning data from the Phase6 vocabulary trainer. The data set contains vocabulary questions in German and English, Spanish, and French as target language and is available in four different variations regarding pre-processing and deduplication. We experiment to fine-tune pre-trained BERT language models on the downstream task of vocabulary evaluation with the proposed MuLVE data set. The results provide outstanding results of > 95.5 accuracy and F2-score. The data set is available on the European Language Grid.