Visual Word Sense Disambiguation with CLIP through Dual-Channel Text Prompting and Image Augmentations

TL;DR

This paper introduces a visual word sense disambiguation framework using CLIP, dual-channel prompts, and image augmentations to improve lexical ambiguity resolution in multimodal contexts.

Contribution

It proposes an interpretable VWSD method leveraging CLIP with dual-channel prompts and robust augmentations, demonstrating improved performance on the SemEval-2023 dataset.

Findings

Enriching embeddings increases MRR from 0.7227 to 0.7590.

Dual-channel prompting offers strong, low-latency performance.

Aggressive image augmentation yields marginal gains.

Abstract

Ambiguity poses persistent challenges in natural language understanding for large language models (LLMs). To better understand how lexical ambiguity can be resolved through the visual domain, we develop an interpretable Visual Word Sense Disambiguation (VWSD) framework. The model leverages CLIP to project ambiguous language and candidate images into a shared multimodal space. We enrich textual embeddings using a dual-channel ensemble of semantic and photo-based prompts with WordNet synonyms, while image embeddings are refined through robust test-time augmentations. We then use cosine similarity to determine the image that best aligns with the ambiguous text. When evaluated on the SemEval-2023 VWSD dataset, enriching the embeddings raises the MRR from 0.7227 to 0.7590 and the Hit Rate from 0.5810 to 0.6220. Ablation studies reveal that dual-channel prompting provides strong, low-latency performance, whereas aggressive image augmentation yields only marginal gains. Additional experiments with WordNet definitions and multilingual prompt ensembles further suggest that noisy external signals tend to dilute semantic specificity, reinforcing the effectiveness of precise, CLIP-aligned prompts for visual word sense disambiguation.