Scaling the Convex Barrier with Sparse Dual Algorithms

TL;DR

This paper introduces two novel dual algorithms that significantly improve the efficiency and tightness of neural network bounding, enabling faster and more accurate verification of complex properties.

Contribution

The paper presents two new dual algorithms that leverage sparsity and active set methods to efficiently solve tighter relaxations for neural network verification.

Findings

Achieved tighter bounds than existing solvers.

Provided significant speed-ups in neural network verification.

Enabled GPU-accelerated parallel computation for bounds calculation.

Abstract

Tight and efficient neural network bounding is crucial to the scaling of neural network verification systems. Many efficient bounding algorithms have been presented recently, but they are often too loose to verify more challenging properties. This is due to the weakness of the employed relaxation, which is usually a linear program of size linear in the number of neurons. While a tighter linear relaxation for piecewise-linear activations exists, it comes at the cost of exponentially many constraints and currently lacks an efficient customized solver. We alleviate this deficiency by presenting two novel dual algorithms: one operates a subgradient method on a small active set of dual variables, the other exploits the sparsity of Frank-Wolfe type optimizers to incur only a linear memory cost. Both methods recover the strengths of the new relaxation: tightness and a linear separation oracle. At the same time, they share the benefits of previous dual approaches for weaker relaxations: massive parallelism, GPU implementation, low cost per iteration and valid bounds at any time. As a consequence, we can obtain better bounds than off-the-shelf solvers in only a fraction of their running time, attaining significant formal verification speed-ups.