REBAR: Reference Ethical Benchmark for Autonomy Readiness

TL;DR

REBAR is a quantitative framework that evaluates autonomous systems' ethical readiness using a computable metric, innovative LLM techniques, and a photorealistic simulation environment.

Contribution

The paper introduces REBAR, a novel benchmark that quantifies ethical performance of autonomous systems with LLM-driven scenario generation and explainability.

Findings

REBAR provides an objective, repeatable ethical benchmark score.

It uses LLMs to assess and explain ethical difficulty of scenarios.

The framework bridges the gap between ethical principles and verifiable autonomy.

Abstract

As autonomous systems grow more advanced, objective metrics to evaluate their ethical and legal compliance are critical for informing end users of their limitations and ensuring accountability of those who misuse them. Current ethical embodied AI frameworks remain mostly qualitative, focusing on system design (through safety guardrails or targeted red teaming), and the realized guardrails often directly disallow unsafe behavior without providing the user with an override or interpretable reason. Instead, there is a need for computable metrics through rigorous testing that allow a user to determine the applicability of the system to the task. To address this gap, we introduce the Reference Ethical Benchmark for Autonomy Readiness (REBAR), a quantitative test and evaluation framework for autonomous systems. REBAR maps operating metrics into a computable Autonomy Readiness Level (ARL) rubric that can quantify ethical performance. Key innovations of the framework include a neuro-symbolic Large Language Model (LLM) approach to calculate and explain the ethical difficulty of scenarios, LLM-driven at-scale generation of test instances, and a versatile, photorealistic simulation environment. By evaluating white-box autonomy solutions through this rigorous testing pipeline, REBAR delivers an objective and repeatable benchmark score, bridging the gap between abstract principles and verifiable, accountable autonomy.