COGNAC at SemEval-2026 Task 5: LLM Ensembles for Human-Level Word Sense Plausibility Rating in Challenging Narratives

TL;DR

This paper presents an ensemble approach using multiple prompting strategies with large language models to evaluate word sense plausibility in narratives, achieving human-level performance and improving subjective semantic evaluation.

Contribution

It introduces a novel ensemble method combining three prompting strategies with LLMs, effectively handling inter-annotator variability in semantic plausibility tasks.

Findings

Ensemble of LLMs achieved 0.88 accuracy and 0.83 Spearman's rho in SemEval-2026 Task 5.

Comparative prompting improved model performance consistently.

Model ensembling enhanced alignment with human judgments in subjective tasks.

Abstract

We describe our system for SemEval-2026 Task 5, which requires rating the plausibility of given word senses of homonyms in short stories on a 5-point Likert scale. Systems are evaluated by the unweighted average of accuracy (within one standard deviation of mean human judgments) and Spearman Rank Correlation. We explore three prompting strategies using multiple closed-source commercial LLMs: (i) a baseline zero-shot setup, (ii) Chain-of-Thought (CoT) style prompting with structured reasoning, and (iii) a comparative prompting strategy for evaluating candidate word senses simultaneously. Furthermore, to account for the substantial inter-annotator variation present in the gold labels, we propose an ensemble setup by averaging model predictions. Our best official system, comprising an ensemble of LLMs across all three prompting strategies, placed 4th on the competition leaderboard with 0.88 accuracy and 0.83 Spearman's rho (0.86 average). Post-competition experiments with additional models further improved this performance to 0.92 accuracy and 0.85 Spearman's rho (0.89 average). We find that comparative prompting consistently improved performance across model families, and model ensembling significantly enhanced alignment with mean human judgments, suggesting that LLM ensembles are especially well suited for subjective semantic evaluation tasks involving multiple annotators.