Almost-Orthogonal Layers for Efficient General-Purpose Lipschitz Networks

TL;DR

This paper introduces Almost-Orthogonal Layers (AOL) for constructing efficient, robust Lipschitz networks with formal guarantees, broad applicability, and simplicity, achieving competitive results in image classification tasks.

Contribution

It proposes a rescaling-based weight parametrization for Lipschitz networks that guarantees a Lipschitz constant of at most 1 and produces near-orthogonal weight matrices, simplifying implementation.

Findings

AOL layers achieve competitive certified robust accuracy.

The method is easier to implement than existing approaches.

AOL layers do not require expensive orthogonalization or inversion steps.

Abstract

It is a highly desirable property for deep networks to be robust against small input changes. One popular way to achieve this property is by designing networks with a small Lipschitz constant. In this work, we propose a new technique for constructing such Lipschitz networks that has a number of desirable properties: it can be applied to any linear network layer (fully-connected or convolutional), it provides formal guarantees on the Lipschitz constant, it is easy to implement and efficient to run, and it can be combined with any training objective and optimization method. In fact, our technique is the first one in the literature that achieves all of these properties simultaneously. Our main contribution is a rescaling-based weight matrix parametrization that guarantees each network layer to have a Lipschitz constant of at most 1 and results in the learned weight matrices to be close to orthogonal. Hence we call such layers almost-orthogonal Lipschitz (AOL). Experiments and ablation studies in the context of image classification with certified robust accuracy confirm that AOL layers achieve results that are on par with most existing methods. Yet, they are simpler to implement and more broadly applicable, because they do not require computationally expensive matrix orthogonalization or inversion steps as part of the network architecture. We provide code at https://github.com/berndprach/AOL.