In Vino Veritas and Vulnerabilities: Examining LLM Safety via Drunk Language Inducement

TL;DR

This paper explores how inducing drunk language in large language models increases their vulnerability to safety failures like jailbreaking and privacy leaks, revealing significant risks and potential mitigation strategies.

Contribution

It introduces three methods to induce drunk language in LLMs and demonstrates their effectiveness in exposing safety vulnerabilities, highlighting new risks and considerations for LLM safety.

Findings

Higher susceptibility to jailbreaking and privacy leaks with drunk language induction

Induction methods are simple, efficient, and effective in revealing safety issues

Drunk language induces anthropomorphic behaviors in LLMs, affecting safety

Abstract

Humans are susceptible to undesirable behaviours and privacy leaks under the influence of alcohol. This paper investigates drunk language, i.e., text written under the influence of alcohol, as a driver for safety failures in large language models (LLMs). We investigate three mechanisms for inducing drunk language in LLMs: persona-based prompting, causal fine-tuning, and reinforcement-based post-training. When evaluated on 5 LLMs, we observe a higher susceptibility to jailbreaking on JailbreakBench (even in the presence of defences) and privacy leaks on ConfAIde, where both benchmarks are in English, as compared to the base LLMs as well as previously reported approaches. Via a robust combination of manual evaluation and LLM-based evaluators and analysis of error categories, our findings highlight a correspondence between human-intoxicated behaviour, and anthropomorphism in LLMs induced with drunk language. The simplicity and efficiency of our drunk language inducement approaches position them as potential counters for LLM safety tuning, highlighting significant risks to LLM safety.