GUARDIAN: Safety Filtering for Systems with Perception Models Subject to Adversarial Attacks

TL;DR

GUARDIAN is a safety filtering framework that guarantees safety for systems with neural network-based state estimators under adversarial attacks by providing formal bounds and adjusting control inputs accordingly.

Contribution

This paper introduces GUARDIAN, a novel safety filtering method that offers formal safety guarantees for systems with NN estimators under adversarial conditions.

Findings

GUARDIAN effectively defends against adversarial attacks.

It provides formal safety guarantees using neural network verification.

Numerical experiments validate its robustness and effectiveness.

Abstract

Safety filtering is an effective method for enforcing constraints in safety-critical systems, but existing methods typically assume perfect state information. This limitation is especially problematic for systems that rely on neural network (NN)-based state estimators, which can be highly sensitive to noise and adversarial input perturbations. We address these problems by introducing GUARDIAN: Guaranteed Uncertainty-Aware Reachability Defense against Adversarial INterference, a safety filtering framework that provides formal safety guarantees for systems with NN-based state estimators. At runtime, GUARDIAN uses neural network verification tools to provide guaranteed bounds on the system's state estimate given possible perturbations to its observation. It then uses a modified Hamilton-Jacobi reachability formulation to construct a safety filter that adjusts the nominal control input based on the verified state bounds and safety constraints. The result is an uncertainty-aware filter that ensures safety despite the system's reliance on an NN estimator with noisy, possibly adversarial, input observations. Theoretical analysis and numerical experiments demonstrate that GUARDIAN effectively defends systems against adversarial attacks that would otherwise lead to a violation of safety constraints.