On the Lipschitz Continuity of Set Aggregation Functions and Neural Networks for Sets

TL;DR

This paper analyzes the Lipschitz continuity of set aggregation functions and neural networks for set-structured data, providing theoretical bounds and empirical validation for their stability and generalization properties.

Contribution

It characterizes Lipschitz continuity of common aggregation functions and neural networks for sets, deriving bounds and highlighting limitations of attention-based methods.

Findings

Aggregation functions are Lipschitz continuous with respect to only one of three distance metrics.

Attention-based aggregation is not Lipschitz continuous under the considered metrics.

Derived upper bounds on neural network Lipschitz constants improve understanding of stability and generalization.

Abstract

The Lipschitz constant of a neural network is connected to several important properties of the network such as its robustness and generalization. It is thus useful in many settings to estimate the Lipschitz constant of a model. Prior work has focused mainly on estimating the Lipschitz constant of multi-layer perceptrons and convolutional neural networks. Here we focus on data modeled as sets or multi-sets of vectors and on neural networks that can handle such data. These models typically apply some permutation invariant aggregation function, such as the sum, mean or max operator, to the input multisets to produce a single vector for each input sample. In this paper, we investigate whether these aggregation functions, along with an attention-based aggregation function, are Lipschitz continuous with respect to three distance functions for unordered multisets, and we compute their Lipschitz constants. In the general case, we find that each aggregation function is Lipschitz continuous with respect to only one of the three distance functions, while the attention-based function is not Lipschitz continuous with respect to any of them. Then, we build on these results to derive upper bounds on the Lipschitz constant of neural networks that can process multisets of vectors, while we also study their stability to perturbations and generalization under distribution shifts. To empirically verify our theoretical analysis, we conduct a series of experiments on datasets from different domains.