Prompting Science Report 3: I'll pay you or I'll kill you -- but will you care?

TL;DR

This report empirically tests common prompting beliefs like tipping and threatening AI models, finding they generally do not improve performance, while prompt variations can significantly influence question-by-question results.

Contribution

It provides rigorous empirical evidence that tipping and threatening do not enhance AI performance, highlighting the importance of prompt design over such tactics.

Findings

Threatening or tipping has no significant effect on benchmark performance.

Prompt variations can significantly alter question-level outcomes.

Simple prompting tactics may be less effective for difficult problems.

Abstract

This is the third in a series of short reports that seek to help business, education, and policy leaders understand the technical details of working with AI through rigorous testing. In this report, we investigate two commonly held prompting beliefs: a) offering to tip the AI model and b) threatening the AI model. Tipping was a commonly shared tactic for improving AI performance and threats have been endorsed by Google Founder Sergey Brin (All-In, May 2025, 8:20) who observed that 'models tend to do better if you threaten them,' a claim we subject to empirical testing here. We evaluate model performance on GPQA (Rein et al. 2024) and MMLU-Pro (Wang et al. 2024). We demonstrate two things: - Threatening or tipping a model generally has no significant effect on benchmark performance. - Prompt variations can significantly affect performance on a per-question level. However, it is hard to know in advance whether a particular prompting approach will help or harm the LLM's ability to answer any particular question. Taken together, this suggests that simple prompting variations might not be as effective as previously assumed, especially for difficult problems. However, as reported previously (Meincke et al. 2025a), prompting approaches can yield significantly different results for individual questions.