Parseval Proximal Neural Networks

TL;DR

This paper introduces Parseval proximal neural networks (PPNNs), which leverage properties of proximity operators and tight frames to create stable, Lipschitz-continuous neural networks with convergence guarantees for applications like adversarial defense.

Contribution

The paper establishes that concatenations of proximity operators with affine operators form new proximity operators, enabling the construction of stable, Lipschitz neural networks based on frame analysis and synthesis.

Findings

PPNNs are Lipschitz networks with Lipschitz constant 1.

PPNNs can be integrated into Plug-and-Play algorithms with convergence guarantees.

Proof-of-concept examples show competitive performance.

Abstract

The aim of this paper is twofold. First, we show that a certain concatenation of a proximity operator with an affine operator is again a proximity operator on a suitable Hilbert space. Second, we use our findings to establish so-called proximal neural networks (PNNs) and stable tight frame proximal neural networks. Let $\mathcal H$ and $\mathcal K$ be real Hilbert spaces, $b\in\mathcal K$ and $T\in\mathcal{B}(\mathcal H,\mathcal K)$ have closed range and Moore-Penrose inverse $T^\dagger$. Based on the well-known characterization of proximity operators by Moreau, we prove that for any proximity operator $\text{Prox}\colon\mathcal K\to\mathcal K$ the operator $T^\dagger\,\text{Prox} (T\cdot +b)$ is a proximity operator on $\mathcal H$ equipped with a suitable norm. In particular, it follows for the frequently applied soft shrinkage operator $\text{Prox} = S_{\lambda}\colon\ell_2 \rightarrow\ell_2$ and any frame analysis operator $T\colon\mathcal H\to\ell_2$ that the frame shrinkage operator $T^\dagger\, S_\lambda\,T$ is a proximity operator on a suitable Hilbert space. The concatenation of proximity operators on $\mathbb R^d$ equipped with different norms establishes a PNN. If the network arises from tight frame analysis or synthesis operators, then it forms an averaged operator. Hence, it has Lipschitz constant 1 and belongs to the class of so-called Lipschitz networks, which were recently applied to defend against adversarial attacks. Moreover, due to its averaging property, PNNs can be used within so-called Plug-and-Play algorithms with convergence guarantee. In case of Parseval frames, we call the networks Parseval proximal neural networks (PPNNs). Then, the involved linear operators are in a Stiefel manifold and corresponding minimization methods can be applied for training. Finally, some proof-of-the concept examples demonstrate the performance of PPNNs.