Can LLMs assist with Ambiguity? A Quantitative Evaluation of various Large Language Models on Word Sense Disambiguation

TL;DR

This paper evaluates the effectiveness of Large Language Models in resolving lexical ambiguity through a novel prompt augmentation approach, significantly improving Word Sense Disambiguation performance in digital communication contexts.

Contribution

It introduces a systematic prompt augmentation method with human-in-the-loop support and few-shot Chain of Thought prompting to enhance LLM-based WSD accuracy.

Findings

Significant performance improvement in WSD tasks.

Effective use of prompt augmentation with POS tagging and sense filtering.

Enhanced interpretation accuracy in social media and digital communication.

Abstract

Ambiguous words are often found in modern digital communications. Lexical ambiguity challenges traditional Word Sense Disambiguation (WSD) methods, due to limited data. Consequently, the efficiency of translation, information retrieval, and question-answering systems is hindered by these limitations. This study investigates the use of Large Language Models (LLMs) to improve WSD using a novel approach combining a systematic prompt augmentation mechanism with a knowledge base (KB) consisting of different sense interpretations. The proposed method incorporates a human-in-loop approach for prompt augmentation where prompt is supported by Part-of-Speech (POS) tagging, synonyms of ambiguous words, aspect-based sense filtering and few-shot prompting to guide the LLM. By utilizing a few-shot Chain of Thought (COT) prompting-based approach, this work demonstrates a substantial improvement in performance. The evaluation was conducted using FEWS test data and sense tags. This research advances accurate word interpretation in social media and digital communication.