Do LLMs Understand Ambiguity in Text? A Case Study in Open-world Question Answering

TL;DR

This paper investigates how well Large Language Models understand ambiguity in open-domain question answering and demonstrates that simple, training-free disambiguation methods can improve their performance.

Contribution

The study introduces and evaluates token-level disambiguation strategies that enhance LLM performance on ambiguous questions without additional training.

Findings

Token-level disambiguation improves accuracy

Simple methods outperform complex ones in some cases

Explicit disambiguation reduces hallucinations and biases

Abstract

Ambiguity in natural language poses significant challenges to Large Language Models (LLMs) used for open-domain question answering. LLMs often struggle with the inherent uncertainties of human communication, leading to misinterpretations, miscommunications, hallucinations, and biased responses. This significantly weakens their ability to be used for tasks like fact-checking, question answering, feature extraction, and sentiment analysis. Using open-domain question answering as a test case, we compare off-the-shelf and few-shot LLM performance, focusing on measuring the impact of explicit disambiguation strategies. We demonstrate how simple, training-free, token-level disambiguation methods may be effectively used to improve LLM performance for ambiguous question answering tasks. We empirically show our findings and discuss best practices and broader impacts regarding ambiguity in LLMs.