Are Dilemmas and Conflicts in LLM Alignment Solvable? A View from Priority Graph

TL;DR

This paper models conflicts in LLM alignment using a priority graph, revealing challenges in achieving stable alignment and proposing a runtime verification method to improve robustness against manipulation.

Contribution

It introduces a novel priority graph framework to analyze LLM conflicts and proposes a runtime verification approach to enhance safety and robustness.

Findings

Priority graphs expose alignment instability and context-dependence.

Vulnerability to priority hacking allows manipulation of LLM outputs.

Runtime verification improves resistance to deceptive contexts.

Abstract

As Large Language Models (LLMs) become more powerful and autonomous, they increasingly face conflicts and dilemmas in many scenarios. We first summarize and taxonomize these diverse conflicts. Then, we model the LLM's preferences to make different choices as a priority graph, where instructions and values are nodes, and the edges represent context-specific priorities determined by the model's output distribution. This graph reveals that a unified stable LLM alignment is very challenging, because the graph is neither static nor necessarily consistent in different contexts. Besides, it also reveals a potential vulnerability: priority hacking, where adversaries can craft deceptive contexts to manipulate the graph and bypass safety alignments. To counter this, we propose a runtime verification mechanism, enabling LLMs to query external sources to ground their context and resist manipulation. While this approach enhances robustness, we also acknowledge that many ethical and value dilemmas are philosophically irreducible, posing a long-term, open challenge for the future of AI alignment.