NCL-UoR at SemEval-2026 Task 5: Embedding-Based Methods, Fine-Tuning, and LLMs for Word Sense Plausibility Rating

TL;DR

This paper compares embedding methods, fine-tuning, and LLM prompting for rating word sense plausibility, finding structured prompting with decision rules most effective.

Contribution

It introduces a structured prompting approach with explicit decision rules that outperforms other methods for word sense plausibility rating.

Findings

Structured prompting outperforms fine-tuning and embedding-based methods.

Prompt design impacts performance more than model size.

Explicit decision rules improve rating calibration.

Abstract

Word sense plausibility rating requires predicting the human-perceived plausibility of a given word sense on a 1-5 scale in the context of short narrative stories containing ambiguous homonyms. This paper systematically compares three approaches: (1) embedding-based methods pairing sentence embeddings with standard regressors, (2) transformer fine-tuning with parameter-efficient adaptation, and (3) large language model (LLM) prompting with structured reasoning and explicit decision rules. The best-performing system employs a structured prompting strategy that decomposes evaluation into narrative components (precontext, target sentence, ending) and applies explicit decision rules for rating calibration. The analysis reveals that structured prompting with decision rules outperforms both fine-tuned models and embedding-based approaches, and that prompt design matters more than model scale for this task.