Talking Nonsense: Probing Large Language Models' Understanding of Adversarial Gibberish Inputs

TL;DR

This paper investigates whether large language models understand gibberish inputs by crafting prompts that elicit coherent responses, revealing insights into their internal mechanisms and robustness to nonsensical prompts.

Contribution

It introduces a systematic method using Greedy Coordinate Gradient optimization to probe LLMs with gibberish inputs, uncovering their behavior and alignment issues.

Findings

Manipulation efficiency varies with prompt length and perplexity

Babel prompts often find lower loss minima than natural prompts

Generating harmful and benign texts from gibberish is similarly feasible

Abstract

Large language models (LLMs) exhibit excellent ability to understand human languages, but do they also understand their own language that appears gibberish to us? In this work we delve into this question, aiming to uncover the mechanisms underlying such behavior in LLMs. We employ the Greedy Coordinate Gradient optimizer to craft prompts that compel LLMs to generate coherent responses from seemingly nonsensical inputs. We call these inputs LM Babel and this work systematically studies the behavior of LLMs manipulated by these prompts. We find that the manipulation efficiency depends on the target text's length and perplexity, with the Babel prompts often located in lower loss minima compared to natural prompts. We further examine the structure of the Babel prompts and evaluate their robustness. Notably, we find that guiding the model to generate harmful texts is not more difficult than into generating benign texts, suggesting lack of alignment for out-of-distribution prompts.