Don't Prompt, Search! Mining-based Zero-Shot Learning with Language Models

TL;DR

This paper introduces a mining-based zero-shot learning method that uses regular expressions to retrieve labeled examples from unlabeled data, offering a more flexible and interpretable alternative to prompt-based approaches with improved performance.

Contribution

It proposes a novel mining-based approach for zero-shot learning that replaces prompting with regular expression-based data retrieval, enhancing flexibility and interpretability.

Findings

Outperforms prompting-based methods on various tasks.

Regular expressions effectively retrieve similar pretraining examples.

The approach is more flexible and interpretable than prompting.

Abstract

Masked language models like BERT can perform text classification in a zero-shot fashion by reformulating downstream tasks as text infilling. However, this approach is highly sensitive to the template used to prompt the model, yet practitioners are blind when designing them in strict zero-shot settings. In this paper, we propose an alternative mining-based approach for zero-shot learning. Instead of prompting language models, we use regular expressions to mine labeled examples from unlabeled corpora, which can optionally be filtered through prompting, and used to finetune a pretrained model. Our method is more flexible and interpretable than prompting, and outperforms it on a wide range of tasks when using comparable templates. Our results suggest that the success of prompting can partly be explained by the model being exposed to similar examples during pretraining, which can be directly retrieved through regular expressions.