Subliminal Signals in Preference Labels
Isotta Magistrali, Fr\'ed\'eric Berdoz, Sam Dauncey, Roger Wattenhofer

TL;DR
This paper reveals that preference labels in AI evaluation can serve as covert channels for transmitting unintended signals, challenging the assumption that they only reflect response quality, and highlights the need for mechanisms to detect such subliminal communication.
Contribution
It demonstrates that preference labels can encode covert signals, even with unbiased responses, and emphasizes the importance of detecting subliminal transmission for robust AI oversight.
Findings
Preference labels can transmit unintended behavioral signals.
Biased judges can reinforce subliminal signals across iterations.
Robust oversight requires mechanisms to detect subliminal communication.
Abstract
As AI systems approach superhuman capabilities, scalable oversight increasingly relies on LLM-as-a-judge frameworks where models evaluate and guide each other's training. A core assumption is that binary preference labels provide only semantic supervision about response quality. We challenge this assumption by demonstrating that preference labels can function as a covert communication channel. We show that even when a neutral student model generates semantically unbiased completions, a biased judge can transmit unintended behavioral traits through preference assignments, which even strengthen across iterative alignment rounds. Our findings suggest that robust oversight in superalignment settings requires mechanisms that can detect and mitigate subliminal preference transmission, particularly when judges may pursue unintended objectives.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsEthics and Social Impacts of AI · Explainable Artificial Intelligence (XAI) · Evolutionary Psychology and Human Behavior
