ARCANE: A Multi-Agent Framework for Interpretable and Configurable Alignment

TL;DR

ARCANE introduces a multi-agent framework that uses natural-language rubrics for interpretable, adaptable reward modeling, enabling better alignment of large language models with stakeholder preferences in complex tasks.

Contribution

The paper presents a novel multi-agent approach that dynamically generates interpretable rubrics for reward modeling, allowing real-time preference shifts without retraining.

Findings

Rubrics are compact and legible, aiding interpretability.

Configurable trade-offs are achievable without retraining.

Rubric-based reward models improve alignment in complex tasks.

Abstract

As agents based on large language models are increasingly deployed to long-horizon tasks, maintaining their alignment with stakeholder preferences becomes critical. Effective alignment in such settings requires reward models that are interpretable so that stakeholders can understand and audit model objectives. Moreover, reward models must be capable of steering agents at interaction time, allowing preference shifts to be incorporated without retraining. We introduce ARCANE, a framework that frames alignment as a multi-agent collaboration problem that dynamically represents stakeholder preferences as natural-language rubrics: weighted sets of verifiable criteria that can be generated on-the-fly from task context. Inspired by utility theory, we formulate rubric learning as a reconstruction problem and apply a regularized Group-Sequence Policy Optimization (GSPO) procedure that balances interpretability, faithfulness, and computational efficiency. Using a corpus of 219 labeled rubrics derived from the GDPVal benchmark, we evaluate ARCANE on challenging tasks requiring multi-step reasoning and tool use. The learned rubrics produce compact, legible evaluations and enable configurable trade-offs (e.g., correctness vs. conciseness) without retraining. Our results show that rubric-based reward models offer a promising path toward interpretable, test-time adaptive alignment for complex, long-horizon AI systems.