Make Interval Bound Propagation great again
Patryk Krukowski, Daniel Wilczak, Jacek Tabor, Anna Bielawska,, Przemys{\l}aw Spurek
TL;DR
This paper improves neural network robustness certification by replacing Interval Bound Propagation with more precise methods like Affine Arithmetic, significantly reducing the wrapping effect and providing tighter bounds.
Contribution
It introduces adapted classical strict computation techniques to neural network certification, overcoming IBP's limitations due to the wrapping effect.
Findings
Affine Arithmetic yields tighter bounds than IBP.
The proposed methods are effective for networks with linear activation.
Bounds are significantly closer to the optimal than previous IBP-based bounds.
Abstract
In various scenarios motivated by real life, such as medical data analysis, autonomous driving, and adversarial training, we are interested in robust deep networks. A network is robust when a relatively small perturbation of the input cannot lead to drastic changes in output (like change of class, etc.). This falls under the broader scope field of Neural Network Certification (NNC). Two crucial problems in NNC are of profound interest to the scientific community: how to calculate the robustness of a given pre-trained network and how to construct robust networks. The common approach to constructing robust networks is Interval Bound Propagation (IBP). This paper demonstrates that IBP is sub-optimal in the first case due to its susceptibility to the wrapping effect. Even for linear activation, IBP gives strongly sub-optimal bounds. Consequently, one should use strategies immune to the…
Peer Reviews
Decision·ICLR 2025 Conference Withdrawn Submission
1. Precisely certifying the robustness of neural networks remains a well-motivated problem. 2. This paper is well-written.
1. Overall, the setting is ad-hoc. The motivation of this work is the wrapping effect, specifically, the precision loss for a sequence randomly orthogonal maps is exponential. This is not a realistic setting. Moreover, the interpretation of Proposition 2.1 in the context of IBP is questionable. 2. Novelty is limited. The mitigation strategies, dubleton arithmetic and affine arithmetic, were already known. This work adapts these two known techniques to the problem. 3. The baseline is weak. The
1. Certification of pre-trained NNs is an important task for many different applications. The paper suggests novel theoretical approaches to achieve tighter bounds. 2. The intuition is well-explained, and the experiments are well-organized and thoroughly analyzed.
1. The method is only applicable for the linear transformation. However, there are various practical models with non-linear activation functions, which make its application is limited. 2. The time-complexity is too high for real applications such as ResNet and transformer-based models.
1. Considering the approximation error in IBP from a different perspective is very interesting to me. 2. The experiment results are strong compared to the original IBP method.
1. The conclusions about IBP wrapping effect has been discussed in many existing works such as [1], with a very similar theoretical result. 2. The proposed method looks very similar to the forward CROWN [2], which uses affine functions to bound the neural networks. Therefore, the novelty of this paper is questionable given these relevant existing works. [1] Fast certified robust training with short warmup. [2] Automatic Perturbation Analysis for Scalable Certified Robustness and Beyond.
The paper is well-structured and easy to read. The proofs are (mostly) easy to follow. Using bounding methods from the field of ODE solvers in Neural Network Certification is an interesting direction. The experiments are well organized and the results are presented and interpreted in insightful ways.
**Lack of discussion on relevant related work** * The first claimed contribution of the paper (theoretical analysis of the wrapping effect in the case of IBP bounds) has been widely discussed in multiple recent works: Shi et al. (2021)[1] show that IBP bounds grow by a factor of $O(\sqrt n)$ after each affine layer, leading to exponential growth for deep networks. Various works [2,3,4,5,6] have proposed methods to compute tighter bounds for certification because IBP was not good enough. Other w
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Wireless Communication Techniques · Advanced Adaptive Filtering Techniques