Concept-Based Dictionary Learning for Inference-Time Safety in Vision Language Action Models

TL;DR

This paper presents a concept-based dictionary learning method for inference-time safety in vision-language action models, effectively reducing unsafe behaviors without retraining, and enhancing interpretability and safety in embodied systems.

Contribution

It introduces a novel, plug-in, model-agnostic safety framework using sparse, interpretable dictionaries to identify and attenuate harmful concepts during inference.

Findings

Achieves over 70% reduction in attack success rates.

Maintains task success while improving safety.

First concept-based safety method for embodied systems.

Abstract

Vision Language Action (VLA) models close the perception action loop by translating multimodal instructions into executable behaviors, but this very capability magnifies safety risks: jailbreaks that merely yield toxic text in LLMs can trigger unsafe physical actions in embodied systems. Existing defenses alignment, filtering, or prompt hardening intervene too late or at the wrong modality, leaving fused representations exploitable. We introduce a concept based dictionary learning framework for inference time safety control. By learning sparse, interpretable dictionaries from hidden activations, our method identifies harmful concept directions and attenuates risky components when the estimated risk exceeds a threshold. Experiments on Libero-Harm, BadRobot, RoboPair, and IS-Bench show that our approach achieves state-of-the-art defense performance, cutting attack success rates by over 70\% while maintaining task success. Crucially, the framework is plug-in and model-agnostic, requiring no retraining and integrating seamlessly with diverse VLAs. To our knowledge, this is the first inference time concept based safety method for embodied systems, advancing both interpretability and safe deployment of VLA models.