The coupling effect of Lipschitz regularization in deep neural networks

TL;DR

This paper explores how Lipschitz regularization influences the robustness and expressiveness of deep neural networks, revealing a coupling effect across layers that impacts their performance under input uncertainties.

Contribution

It demonstrates the coupling effect of Lipschitz regularization on network weights and its tradeoff between robustness and expressiveness, providing insights for better regularization practices.

Findings

Lipschitz regularization induces a coupling effect across network layers.

There is a tradeoff between robustness and expressiveness due to this coupling.

Proper implementation of Lipschitz regularization is crucial to balance robustness and expressiveness.

Abstract

We investigate robustness of deep feed-forward neural networks when input data are subject to random uncertainties. More specifically, we consider regularization of the network by its Lipschitz constant and emphasize its role. We highlight the fact that this regularization is not only a way to control the magnitude of the weights but has also a coupling effect on the network weights accross the layers. We claim and show evidence on a dataset that this coupling effect brings a tradeoff between robustness and expressiveness of the network. This suggests that Lipschitz regularization should be carefully implemented so as to maintain coupling accross layers.