Certified Robust Models with Slack Control and Large Lipschitz Constants

TL;DR

This paper introduces a Calibrated Lipschitz-Margin Loss (CLL) that enhances certified robustness of models by balancing Lipschitz constraints and accuracy, outperforming existing methods on multiple datasets.

Contribution

The paper proposes CLL, a novel loss function that calibrates margin and Lipschitz constant, enabling better robustness and accuracy trade-offs in neural networks.

Findings

CLL improves certified robustness on CIFAR-10, CIFAR-100, Tiny-ImageNet.

Models with CLL outperform existing losses that do not calibrate margin and Lipschitz constant.

CIL enables use of smaller models without strict Lipschitz constraints.

Abstract

Despite recent success, state-of-the-art learning-based models remain highly vulnerable to input changes such as adversarial examples. In order to obtain certifiable robustness against such perturbations, recent work considers Lipschitz-based regularizers or constraints while at the same time increasing prediction margin. Unfortunately, this comes at the cost of significantly decreased accuracy. In this paper, we propose a Calibrated Lipschitz-Margin Loss (CLL) that addresses this issue and improves certified robustness by tackling two problems: Firstly, commonly used margin losses do not adjust the penalties to the shrinking output distribution; caused by minimizing the Lipschitz constant $K$. Secondly, and most importantly, we observe that minimization of $K$ can lead to overly smooth decision functions. This limits the model's complexity and thus reduces accuracy. Our CLL addresses these issues by explicitly calibrating the loss w.r.t. margin and Lipschitz constant, thereby establishing full control over slack and improving robustness certificates even with larger Lipschitz constants. On CIFAR-10, CIFAR-100 and Tiny-ImageNet, our models consistently outperform losses that leave the constant unattended. On CIFAR-100 and Tiny-ImageNet, CLL improves upon state-of-the-art deterministic $L_2$ robust accuracies. In contrast to current trends, we unlock potential of much smaller models without $K=1$ constraints.