Via Negativa for AI Alignment: Why Negative Constraints Are Structurally Superior to Positive Preferences

TL;DR

This paper argues that negative constraints are structurally superior to positive preferences for AI alignment because they encode clear, verifiable prohibitions, leading to more stable and effective alignment strategies.

Contribution

It provides a unified theoretical account explaining why negative signals outperform positive preferences in AI alignment, based on their structural asymmetry and epistemological foundations.

Findings

Negative-only feedback methods match or exceed RLHF performance.

Negative constraints lead to stable boundaries for model behavior.

Preference-based methods are prone to sycophancy and surface correlations.

Abstract

Recent empirical results have demonstrated that training large language models (LLMs) with negative-only feedback can match or exceed standard reinforcement learning from human feedback (RLHF). Negative Sample Reinforcement achieves parity with PPO on mathematical reasoning; Distributional Dispreference Optimization trains effectively using only dispreferred samples; and Constitutional AI outperforms pure RLHF on harmlessness benchmarks. Yet no unified theoretical account explains why negative signals are so effective. This paper proposes such an account: positive preferences and negative constraints are structurally asymmetric. Positive preferences ("which is better") encode continuously coupled, context-dependent human values that cannot be exhaustively specified -- leading models to learn surface correlates such as agreement with the user (sycophancy). Negative constraints ("what is wrong") encode discrete, finite, independently verifiable prohibitions that can converge to a stable boundary. This asymmetry -- rooted in Popper's falsification logic and the epistemology of negative knowledge -- explains both the sycophancy failure of preference-based RLHF and the surprising effectiveness of negative-signal methods. We argue that alignment research should shift its center of gravity from "learning what humans prefer" to "learning what humans reject," and offer testable predictions for this framework.