An Empirical Study on Few-shot Knowledge Probing for Pretrained Language Models

TL;DR

This paper investigates the effectiveness of few-shot learning approaches for probing world knowledge in pretrained language models, introducing a new dataset for 2-hop relations and demonstrating that simple fine-tuning methods outperform prompt engineering in low-data scenarios.

Contribution

It compares various few-shot knowledge probing methods, introduces TREx-2p dataset for 2-hop relations, and shows simple bias fine-tuning surpasses prompt engineering techniques.

Findings

Few-shot examples significantly improve probing performance.

Fine-tuning bias vectors outperforms prompt-engineering methods.

The new TREx-2p dataset enables evaluation of 2-hop relations.

Abstract

Prompt-based knowledge probing for 1-hop relations has been used to measure how much world knowledge is stored in pretrained language models. Existing work uses considerable amounts of data to tune the prompts for better performance. In this work, we compare a variety of approaches under a few-shot knowledge probing setting, where only a small number (e.g., 10 or 20) of example triples are available. In addition, we create a new dataset named TREx-2p, which contains 2-hop relations. We report that few-shot examples can strongly boost the probing performance for both 1-hop and 2-hop relations. In particular, we find that a simple-yet-effective approach of finetuning the bias vectors in the model outperforms existing prompt-engineering methods. Our dataset and code are available at \url{https://github.com/cloudygoose/fewshot_lama}.