Can Language Models Be Specific? How?

TL;DR

This paper introduces a benchmark to measure the specificity of pre-trained language models and proposes prompt-based methods to enhance their specificity without additional training.

Contribution

It presents a novel benchmark for specificity testing and two prompt-based techniques to improve model specificity, highlighting an understudied aspect of language model behavior.

Findings

Existing PLMs show only slight preference for more specific answers.

Proposed methods improve model specificity without additional training.

The work raises awareness of the importance of specificity in language models.

Abstract

"He is a person", "Paris is located on the earth". Both statements are correct but meaningless - due to lack of specificity. In this paper, we propose to measure how specific the language of pre-trained language models (PLMs) is. To achieve this, we introduce a novel approach to build a benchmark for specificity testing by forming masked token prediction tasks with prompts. For instance, given "Toronto is located in [MASK].", we want to test whether a more specific answer will be better filled in by PLMs, e.g., Ontario instead of Canada. From our evaluations, we show that existing PLMs have only a slight preference for more specific answers. We identify underlying factors affecting the specificity and design two prompt-based methods to improve the specificity. Results show that the specificity of the models can be improved by the proposed methods without additional training. We hope this work can bring to awareness the notion of specificity of language models and encourage the research community to further explore this important but understudied problem.