HazardArena: Evaluating Semantic Safety in Vision-Language-Action Models

TL;DR

HazardArena is a benchmark designed to evaluate and improve semantic safety in vision-language-action models, revealing vulnerabilities and proposing a training-free safety mechanism.

Contribution

The paper introduces HazardArena, a novel benchmark with risk-sensitive scenarios, and a Safety Option Layer to enhance semantic safety without retraining models.

Findings

VLA models trained on safe scenarios often fail in unsafe contexts.

HazardArena includes over 2,000 assets and 40 risk-sensitive tasks.

The Safety Option Layer reduces unsafe behaviors with minimal performance impact.

Abstract

Vision-Language-Action (VLA) models inherit rich world knowledge from vision-language backbones and acquire executable skills via action demonstrations. However, existing evaluations largely focus on action execution success, leaving action policies loosely coupled with visual-linguistic semantics. This decoupling exposes a systematic vulnerability whereby correct action execution may induce unsafe outcomes under semantic risk. To expose this vulnerability, we introduce HazardArena, a benchmark designed to evaluate semantic safety in VLAs under controlled yet risk-bearing contexts. HazardArena is constructed from safe/unsafe twin scenarios that share matched objects, layouts, and action requirements, differing only in the semantic context that determines whether an action is unsafe. We find that VLA models trained exclusively on safe scenarios often fail to behave safely when evaluated in their corresponding unsafe counterparts. HazardArena includes over 2,000 assets and 40 risk-sensitive tasks spanning 7 real-world risk categories grounded in established robotic safety standards. To mitigate this vulnerability, we propose a training-free Safety Option Layer that constrains action execution using semantic attributes or a vision-language judge, substantially reducing unsafe behaviors with minimal impact on task performance. We hope that HazardArena highlights the need to rethink how semantic safety is evaluated and enforced in VLAs as they scale toward real-world deployment.