Constraint-Aware Refinement for Safety Verification of Neural Feedback Loops

TL;DR

This paper introduces CARV, a refinement method that improves safety verification of neural feedback loops by reducing conservativeness of reachability analysis, enabling more efficient and accurate safety guarantees in control systems.

Contribution

The paper proposes a novel constraint-aware refinement strategy (CARV) that enhances reachability analysis for neural feedback loops, making safety verification more efficient and less conservative.

Findings

CARV outperforms existing methods in verification speed and memory usage.

It successfully verifies safety where other approaches fail or are significantly slower.

CARV reduces conservativeness of reachability analysis, enabling verification over complex systems.

Abstract

Neural networks (NNs) are becoming increasingly popular in the design of control pipelines for autonomous systems. However, since the performance of NNs can degrade in the presence of out-of-distribution data or adversarial attacks, systems that have NNs in their control pipelines, i.e., neural feedback loops (NFLs), need safety assurances before they can be applied in safety-critical situations. Reachability analysis offers a solution to this problem by calculating reachable sets that bound the possible future states of an NFL and can be checked against dangerous regions of the state space to verify that the system does not violate safety constraints. Since exact reachable sets are generally intractable to calculate, reachable set over approximations (RSOAs) are typically used. The problem with RSOAs is that they can be overly conservative, making it difficult to verify the satisfaction of safety constraints, especially over long time horizons or for highly nonlinear NN control policies. Refinement strategies such as partitioning or symbolic propagation are typically used to limit the conservativeness of RSOAs, but these approaches come with a high computational cost and often can only be used to verify safety for simple reachability problems. This paper presents Constraint-Aware Refinement for Verification (CARV): an efficient refinement strategy that reduces the conservativeness of RSOAs by explicitly using the safety constraints on the NFL to refine RSOAs only where necessary. We demonstrate that CARV can verify the safety of an NFL where other approaches either fail or take up to 60x longer and 40x the memory.