On the Limitations of Representing Functions on Sets

TL;DR

This paper investigates the limitations of using summation-based permutation-invariant functions for set inputs, showing that continuous mappings require the latent space dimension to grow with set size for universal representation.

Contribution

It proves that continuous models like neural networks impose a lower bound on latent space dimension proportional to maximum set size, challenging prior conjectures.

Findings

Summation-based functions require high-dimensional latent spaces for large sets.

Discontinuous mappings can bypass dimensionality constraints but are of limited practical use.

Universal set function representation needs latent dimension at least as large as the maximum set size.

Abstract

Recent work on the representation of functions on sets has considered the use of summation in a latent space to enforce permutation invariance. In particular, it has been conjectured that the dimension of this latent space may remain fixed as the cardinality of the sets under consideration increases. However, we demonstrate that the analysis leading to this conjecture requires mappings which are highly discontinuous and argue that this is only of limited practical use. Motivated by this observation, we prove that an implementation of this model via continuous mappings (as provided by e.g. neural networks or Gaussian processes) actually imposes a constraint on the dimensionality of the latent space. Practical universal function representation for set inputs can only be achieved with a latent dimension at least the size of the maximum number of input elements.