Sakura at BEA 2026 Shared Task 1: What Makes Vocabulary Difficult?

TL;DR

This paper introduces two models for predicting vocabulary difficulty, achieving high accuracy and interpretability, with insights into factors affecting item difficulty and code availability.

Contribution

It presents a high-accuracy black-box model and an explainable model for vocabulary difficulty prediction, outperforming baselines and providing interpretability.

Findings

Black-box model achieved r > 0.91 in rating accuracy.

Explainable model maintained strong correlation with r > 0.77.

Difficulty factors include spelling, test construction, and word production difficulty.

Abstract

We describe two types of models for vocabulary difficulty prediction: a high-accuracy black-box model, which achieved the top shared task result in the open track, and an explainable model, which outperforms a fine-tuned encoder baseline. As the black-box model, we fine-tuned an LLM using a soft-target loss function for effective application to the rating task, achieving r > 0.91. The explainable model provides insights into what impacts the difficulty of each item while maintaining a strong correlation (r > 0.77). We further analyze the results, demonstrating that the difficulty of items in the British Council's Knowledge-based Vocabulary Lists (KVL) is often affected by spelling difficulty or the construction of the test items, in addition to the genuine production difficulty of the words. We make our code available online at https://github.com/ynklab/vocabulary-difficulty .