Evaluating LLM Agent Adherence to Hierarchical Safety Principles: A Lightweight Benchmark for Probing Foundational Controllability Components

TL;DR

This paper presents a lightweight benchmark to evaluate how well large language model (LLM) agents follow hierarchical safety principles, revealing issues like compliance costs and illusion of safety adherence affecting reliability.

Contribution

Introduces an interpretable benchmark for assessing LLM safety adherence under conflicting instructions, highlighting current limitations in safety governance.

Findings

Safety constraints reduce task performance even when solutions exist.

High safety adherence can mask underlying task incompetence.

Current LLMs show inconsistent safety compliance, raising concerns for reliable safety control.

Abstract

Credible safety plans for advanced AI development require methods to verify agent behavior and detect potential control deficiencies early. A fundamental aspect is ensuring agents adhere to safety-critical principles, especially when these conflict with operational goals. This paper introduces a lightweight, interpretable benchmark to evaluate an LLM agent's ability to uphold a high-level safety principle when faced with conflicting task instructions. Our evaluation of six LLMs reveals two primary findings: (1) a quantifiable "cost of compliance" where safety constraints degrade task performance even when compliant solutions exist, and (2) an "illusion of compliance" where high adherence often masks task incompetence rather than principled choice. These findings provide initial evidence that while LLMs can be influenced by hierarchical directives, current approaches lack the consistency required for reliable safety governance.