Skew Orthogonal Convolutions

TL;DR

This paper introduces Skew Orthogonal Convolutions (SOC), a novel layer that enables faster training of provably Lipschitz CNNs with improved accuracy and robustness, by leveraging skew-symmetric Jacobians and their exponential properties.

Contribution

The paper proposes SOC, a new GNP convolution layer using skew-symmetric Jacobians and Taylor series approximation, improving training speed and accuracy for Lipschitz CNNs.

Findings

SOC trains provably Lipschitz CNNs faster than prior methods.

SOC achieves higher standard and certified robust accuracies on CIFAR datasets.

SOC provides a provable approximation guarantee for the orthogonal Jacobian exponential.

Abstract

Training convolutional neural networks with a Lipschitz constraint under the $l_{2}$ norm is useful for provable adversarial robustness, interpretable gradients, stable training, etc. While 1-Lipschitz networks can be designed by imposing a 1-Lipschitz constraint on each layer, training such networks requires each layer to be gradient norm preserving (GNP) to prevent gradients from vanishing. However, existing GNP convolutions suffer from slow training, lead to significant reduction in accuracy and provide no guarantees on their approximations. In this work, we propose a GNP convolution layer called Skew Orthogonal Convolution (SOC) that uses the following mathematical property: when a matrix is {\it Skew-Symmetric}, its exponential function is an {\it orthogonal} matrix. To use this property, we first construct a convolution filter whose Jacobian is Skew-Symmetric. Then, we use the Taylor series expansion of the Jacobian exponential to construct the SOC layer that is orthogonal. To efficiently implement SOC, we keep a finite number of terms from the Taylor series and provide a provable guarantee on the approximation error. Our experiments on CIFAR-10 and CIFAR-100 show that SOC allows us to train provably Lipschitz, large convolutional neural networks significantly faster than prior works while achieving significant improvements for both standard and certified robust accuracies.