BarrierBench: Evaluating Large Language Models for Safety Verification in Dynamical Systems

TL;DR

This paper introduces BarrierBench, a benchmark for evaluating large language models in synthesizing safety barrier certificates for dynamical systems, demonstrating over 90% success rate with an LLM-based framework.

Contribution

The work presents a novel LLM-driven framework for barrier certificate synthesis and introduces BarrierBench, a comprehensive benchmark for evaluating such methods in diverse dynamical systems.

Findings

LLM-based framework achieves over 90% success in generating valid safety certificates.

BarrierBench provides a diverse set of 100 dynamical systems for evaluation.

Retrieval-augmented generation improves the reliability of LLM-guided synthesis.

Abstract

Safety verification of dynamical systems via barrier certificates is essential for ensuring correctness in autonomous applications. Synthesizing these certificates involves discovering mathematical functions with current methods suffering from poor scalability, dependence on carefully designed templates, and exhaustive or incremental function-space searches. They also demand substantial manual expertise--selecting templates, solvers, and hyperparameters, and designing sampling strategies--requiring both theoretical and practical knowledge traditionally shared through linguistic reasoning rather than formalized methods. This motivates a key question: can such expert reasoning be captured and operationalized by language models? We address this by introducing an LLM-based agentic framework for barrier certificate synthesis. The framework uses natural language reasoning to propose, refine, and validate candidate certificates, integrating LLM-driven template discovery with SMT-based verification, and supporting barrier-controller co-synthesis to ensure consistency between safety certificates and controllers. To evaluate this capability, we introduce BarrierBench, a benchmark of 100 dynamical systems spanning linear, nonlinear, discrete-time, and continuous-time settings. Our experiments assess not only the effectiveness of LLM-guided barrier synthesis but also the utility of retrieval-augmented generation and agentic coordination strategies in improving its reliability and performance. Across these tasks, the framework achieves more than 90% success in generating valid certificates. By releasing BarrierBench and the accompanying toolchain, we aim to establish a community testbed for advancing the integration of language-based reasoning with formal verification in dynamical systems. The benchmark is publicly available at https://hycodev.com/dataset/barrierbench