Funzac at CoMeDi Shared Task: Modeling Annotator Disagreement from Word-In-Context Perspectives

TL;DR

This paper investigates annotator disagreement in Word-in-Context tasks by developing methods that incorporate enriched, task-specific features and embeddings, showing improved performance over baseline approaches.

Contribution

The paper introduces three novel methods for modeling annotator disagreement in WiC tasks, including feature enrichment, embedding transformation, and ensemble classifiers, advancing understanding of semantic judgment variability.

Findings

Enriched features improve disagreement modeling performance.

Task-specific embedding transformations enhance representation quality.

Methods are competitive with state-of-the-art in shared task evaluations.

Abstract

In this work, we evaluate annotator disagreement in Word-in-Context (WiC) tasks exploring the relationship between contextual meaning and disagreement as part of the CoMeDi shared task competition. While prior studies have modeled disagreement by analyzing annotator attributes with single-sentence inputs, this shared task incorporates WiC to bridge the gap between sentence-level semantic representation and annotator judgment variability. We describe three different methods that we developed for the shared task, including a feature enrichment approach that combines concatenation, element-wise differences, products, and cosine similarity, Euclidean and Manhattan distances to extend contextual embedding representations, a transformation by Adapter blocks to obtain task-specific representations of contextual embeddings, and classifiers of varying complexities, including ensembles. The comparison of our methods demonstrates improved performance for methods that include enriched and task-specfic features. While the performance of our method falls short in comparison to the best system in subtask 1 (OGWiC), it is competitive to the official evaluation results in subtask 2 (DisWiC).