First Three Years of the International Verification of Neural Networks Competition (VNN-COMP)

TL;DR

This paper reviews the first three years of VNN-COMP, an annual competition that evaluates tools for verifying neural network properties across various applications, highlighting trends and future directions.

Contribution

It provides a comprehensive summary and analysis of VNN-COMP's evolution, results, and challenges over three years, offering insights into the state of neural network verification.

Findings

Increasing participation and tool diversity over years

Emerging trends in verification techniques and benchmarks

Identified challenges and future research directions

Abstract

This paper presents a summary and meta-analysis of the first three iterations of the annual International Verification of Neural Networks Competition (VNN-COMP) held in 2020, 2021, and 2022. In the VNN-COMP, participants submit software tools that analyze whether given neural networks satisfy specifications describing their input-output behavior. These neural networks and specifications cover a variety of problem classes and tasks, corresponding to safety and robustness properties in image classification, neural control, reinforcement learning, and autonomous systems. We summarize the key processes, rules, and results, present trends observed over the last three years, and provide an outlook into possible future developments.