Prompting GPT-3 To Be Reliable

TL;DR

This paper proposes simple prompting techniques to enhance GPT-3's reliability across generalization, bias reduction, calibration, and factual accuracy, making it more dependable for real-world applications.

Contribution

It introduces effective prompts that improve GPT-3's reliability in multiple facets, including out-of-distribution generalization, social bias mitigation, calibration, and factual knowledge updating.

Findings

GPT-3 with prompts outperforms smaller models on reliability metrics

Prompts reduce social biases and improve factual accuracy

Calibration of output probabilities is enhanced

Abstract

Large language models (LLMs) show impressive abilities via few-shot prompting. Commercialized APIs such as OpenAI GPT-3 further increase their use in real-world language applications. However, the crucial problem of how to improve the reliability of GPT-3 is still under-explored. While reliability is a broad and vaguely defined term, we decompose reliability into four main facets that correspond to the existing framework of ML safety and are well-recognized to be important: generalizability, social biases, calibration, and factuality. Our core contribution is to establish simple and effective prompts that improve GPT-3's reliability as it: 1) generalizes out-of-distribution, 2) balances demographic distribution and uses natural language instructions to reduce social biases, 3) calibrates output probabilities, and 4) updates the LLM's factual knowledge and reasoning chains. With appropriate prompts, GPT-3 is more reliable than smaller-scale supervised models on all these facets. We release all processed datasets, evaluation scripts, and model predictions. Our systematic empirical study not only sheds new insights on the reliability of prompting LLMs, but more importantly, our prompting strategies can help practitioners more reliably use LLMs like GPT-3.