AutoSpec: Automated Generation of Neural Network Specifications

TL;DR

AutoSpec is a novel framework that automatically generates accurate and comprehensive specifications for neural networks, improving safety and robustness in learning-augmented systems, and establishing new metrics for evaluation.

Contribution

AutoSpec introduces the first automated method for generating neural network specifications and proposes metrics for their assessment, surpassing manual approaches.

Findings

AutoSpec outperforms human-defined specifications.

AutoSpec surpasses baseline approaches in accuracy and coverage.

The framework is effective across multiple application domains.

Abstract

The increasing adoption of neural networks in learning-augmented systems highlights the importance of model safety and robustness, particularly in safety-critical domains. Despite progress in the formal verification of neural networks, current practices require users to manually define model specifications -- properties that dictate expected model behavior in various scenarios. This manual process, however, is prone to human error, limited in scope, and time-consuming. In this paper, we introduce AutoSpec, the first framework to automatically generate comprehensive and accurate specifications for neural networks in learning-augmented systems. We also propose the first set of metrics for assessing the accuracy and coverage of model specifications, establishing a benchmark for future comparisons. Our evaluation across four distinct applications shows that AutoSpec outperforms human-defined specifications as well as two baseline approaches introduced in this study.