When to Speak, When to Abstain: Contrastive Decoding with Abstention

TL;DR

This paper introduces Contrastive Decoding with Abstention (CDA), a decoding method enabling large language models to generate responses when relevant knowledge is available and abstain otherwise, improving robustness and trustworthiness.

Contribution

The paper proposes a novel, training-free decoding approach that allows LLMs to selectively generate or abstain based on knowledge relevance, addressing robustness issues in knowledge-limited scenarios.

Findings

CDA effectively balances generation and abstention in various scenarios.

CDA improves the reliability and trustworthiness of LLM outputs.

Extensive experiments validate CDA's ability to handle knowledge access challenges.

Abstract

Large Language Models (LLMs) demonstrate exceptional performance across diverse tasks by leveraging pre-trained (i.e., parametric) and external (i.e., contextual) knowledge. While substantial efforts have been made to enhance the utilization of both forms of knowledge, situations in which models lack relevant information remain underexplored. To investigate this challenge, we first present a controlled testbed featuring four distinct knowledge access scenarios, including the aforementioned edge case, revealing that conventional LLM usage exhibits insufficient robustness in handling all instances. Addressing this limitation, we propose Contrastive Decoding with Abstention (CDA), a novel training-free decoding method that allows LLMs to generate responses when relevant knowledge is available and to abstain otherwise. CDA estimates the relevance of both knowledge sources for a given input, adaptively deciding which type of information to prioritize and which to exclude. Through extensive experiments, we demonstrate that CDA can effectively perform accurate generation and abstention simultaneously, enhancing reliability and preserving user trust.