Ukrainian Visual Word Sense Disambiguation Benchmark

TL;DR

This paper introduces a Ukrainian Visual Word Sense Disambiguation benchmark to evaluate multilingual models' ability to identify the correct image representation of ambiguous words with minimal context, highlighting a performance gap with English.

Contribution

It presents the first Ukrainian Visual-WSD benchmark, adapting a methodology from other languages, and evaluates multilingual models, revealing significant performance differences.

Findings

All models underperform compared to the CLIP baseline.

Models perform worse on Ukrainian than on English.

The benchmark enables cross-language performance comparison.

Abstract

This study presents a benchmark for evaluating the Visual Word Sense Disambiguation (Visual-WSD) task in Ukrainian. The main goal of the Visual-WSD task is to identify, with minimal contextual information, the most appropriate representation of a given ambiguous word from a set of ten images. To construct this benchmark, we followed a methodology similar to that proposed by (CITATION), who previously introduced benchmarks for the Visual-WSD task in English, Italian, and Farsi. This approach allows us to incorporate the Ukrainian benchmark into a broader framework for cross-language model performance comparisons. We collected the benchmark data semi-automatically and refined it with input from domain experts. We then assessed eight multilingual and multimodal large language models using this benchmark. All tested models performed worse than the zero-shot CLIP-based baseline model (CITATION) used by (CITATION) for the English Visual-WSD task. Our analysis revealed a significant performance gap in the Visual-WSD task between Ukrainian and English.