Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Models

TL;DR

This paper introduces a standardized black-box evaluation method and dataset to assess Large Language Models' ability to abstain from answering uncertain questions, highlighting current limitations and potential improvements.

Contribution

It presents a new evaluation framework, dataset, and confusion matrix for assessing abstention abilities of LLMs, applicable to black-box models, and explores prompting strategies to improve abstention performance.

Findings

GPT-4 and Mixtral 8x22b struggle with abstention

Strict prompting and Chain-of-Thought improve abstention ability

Proposed AUCM offers a structured evaluation approach

Abstract

Abstention Ability (AA) is a critical aspect of Large Language Model (LLM) reliability, referring to an LLM's capability to withhold responses when uncertain or lacking a definitive answer, without compromising performance. Although previous studies have attempted to improve AA, they lack a standardised evaluation method and remain unsuitable for black-box models where token prediction probabilities are inaccessible. This makes comparative analysis challenging, especially for state-of-the-art closed-source commercial LLMs. This paper bridges this gap by introducing a black-box evaluation approach and a new dataset, Abstain-QA, crafted to rigorously assess AA across varied question types (answerable and unanswerable), domains (well-represented and under-represented), and task types (fact centric and reasoning). We also propose a new confusion matrix, the ''Answerable-Unanswerable Confusion Matrix'' (AUCM) which serves as the basis for evaluating AA, by offering a structured and precise approach for assessment. Finally, we explore the impact of three prompting strategies-Strict Prompting, Verbal Confidence Thresholding, and Chain-of-Thought (CoT)-on improving AA. Our results indicate that even powerful models like GPT-4, Mixtral 8x22b encounter difficulties with abstention; however, strategic approaches such as Strict prompting and CoT can enhance this capability.