UAlberta at SemEval-2023 Task 1: Context Augmentation and Translation for Multilingual Visual Word Sense Disambiguation

TL;DR

The paper introduces a novel multilingual visual word sense disambiguation system that combines gloss retrieval, context augmentation with language models, and image-based methods, achieving competitive results in SemEval-2023.

Contribution

It presents a new algorithm integrating BabelNet glosses, context augmentation, and image techniques for improved V-WSD performance.

Findings

Substantial accuracy improvements with context augmentation.

Competitive ranking among 56 teams in SemEval-2023.

Effective use of image generation and segmentation methods.

Abstract

We describe the systems of the University of Alberta team for the SemEval-2023 Visual Word Sense Disambiguation (V-WSD) Task. We present a novel algorithm that leverages glosses retrieved from BabelNet, in combination with text and image encoders. Furthermore, we compare language-specific encoders against the application of English encoders to translated texts. As the contexts given in the task datasets are extremely short, we also experiment with augmenting these contexts with descriptions generated by a language model. This yields substantial improvements in accuracy. We describe and evaluate additional V-WSD methods which use image generation and text-conditioned image segmentation. Overall, the results of our official submission rank us 18 out of 56 teams. Some of our unofficial results are even better than the official ones. Our code is publicly available at https://github.com/UAlberta-NLP/v-wsd.