Lipschitz-aware Linearity Grafting for Certified Robustness

TL;DR

This paper introduces a theoretical and practical method called Lipschitz-aware linearity grafting that reduces approximation errors in neural networks, leading to tighter local Lipschitz constants and improved certified robustness against adversarial attacks.

Contribution

It provides the first theoretical analysis of how linearity grafting enhances certified robustness and proposes a new method to tighten local Lipschitz constants without certified training.

Findings

Linearity grafting improves certified robustness by tightening local Lipschitz constants.

Theoretical analysis explains how grafting reduces approximation errors.

Experiments show enhanced robustness through linearity grafting.

Abstract

Lipschitz constant is a fundamental property in certified robustness, as smaller values imply robustness to adversarial examples when a model is confident in its prediction. However, identifying the worst-case adversarial examples is known to be an NP-complete problem. Although over-approximation methods have shown success in neural network verification to address this challenge, reducing approximation errors remains a significant obstacle. Furthermore, these approximation errors hinder the ability to obtain tight local Lipschitz constants, which are crucial for certified robustness. Originally, grafting linearity into non-linear activation functions was proposed to reduce the number of unstable neurons, enabling scalable and complete verification. However, no prior theoretical analysis has explained how linearity grafting improves certified robustness. We instead consider linearity grafting primarily as a means of eliminating approximation errors rather than reducing the number of unstable neurons, since linear functions do not require relaxation. In this paper, we provide two theoretical contributions: 1) why linearity grafting improves certified robustness through the lens of the $l_\infty$ local Lipschitz constant, and 2) grafting linearity into non-linear activation functions, the dominant source of approximation errors, yields a tighter local Lipschitz constant. Based on these theoretical contributions, we propose a Lipschitz-aware linearity grafting method that removes dominant approximation errors, which are crucial for tightening the local Lipschitz constant, thereby improving certified robustness, even without certified training. Our extensive experiments demonstrate that grafting linearity into these influential activations tightens the $l_\infty$ local Lipschitz constant and enhances certified robustness.