Automated Design of Linear Bounding Functions for Sigmoidal Nonlinearities in Neural Networks

TL;DR

This paper introduces a new parameter search method to enhance linear bounds for Sigmoid and Tanh functions, significantly improving neural network robustness verification accuracy.

Contribution

It presents a novel parameter search approach for better linear bounding of nonlinearities, advancing verification techniques for general activation functions.

Findings

Improves average global lower bound by 25%

Enhances robustness verification accuracy for Sigmoid and Tanh

Outperforms current state-of-the-art methods

Abstract

The ubiquity of deep learning algorithms in various applications has amplified the need for assuring their robustness against small input perturbations such as those occurring in adversarial attacks. Existing complete verification techniques offer provable guarantees for all robustness queries but struggle to scale beyond small neural networks. To overcome this computational intractability, incomplete verification methods often rely on convex relaxation to over-approximate the nonlinearities in neural networks. Progress in tighter approximations has been achieved for piecewise linear functions. However, robustness verification of neural networks for general activation functions (e.g., Sigmoid, Tanh) remains under-explored and poses new challenges. Typically, these networks are verified using convex relaxation techniques, which involve computing linear upper and lower bounds of the nonlinear activation functions. In this work, we propose a novel parameter search method to improve the quality of these linear approximations. Specifically, we show that using a simple search method, carefully adapted to the given verification problem through state-of-the-art algorithm configuration techniques, improves the average global lower bound by 25% on average over the current state of the art on several commonly used local robustness verification benchmarks.