FuocChuVIP123 at CoMeDi Shared Task: Disagreement Ranking with XLM-Roberta Sentence Embeddings and Deep Neural Regression

TL;DR

This paper introduces a system for disagreement ranking in multilingual contexts using XLM-Roberta sentence embeddings and deep neural regression, focusing on predicting annotator judgment differences.

Contribution

It proposes a novel approach that predicts pairwise disagreement levels directly, diverging from traditional gold label aggregation methods.

Findings

Achieved competitive Spearman correlation scores.

Highlighted the importance of robust embeddings and model architecture.

Demonstrated effective handling of judgment differences in multilingual data.

Abstract

This paper presents results of our system for CoMeDi Shared Task, focusing on Subtask 2: Disagreement Ranking. Our system leverages sentence embeddings generated by the paraphrase-xlm-r-multilingual-v1 model, combined with a deep neural regression model incorporating batch normalization and dropout for improved generalization. By predicting the mean of pairwise judgment differences between annotators, our method explicitly targets disagreement ranking, diverging from traditional "gold label" aggregation approaches. We optimized our system with a customized architecture and training procedure, achieving competitive performance in Spearman correlation against mean disagreement labels. Our results highlight the importance of robust embeddings, effective model architecture, and careful handling of judgment differences for ranking disagreement in multilingual contexts. These findings provide insights into the use of contextualized representations for ordinal judgment tasks and open avenues for further refinement of disagreement prediction models.