On the Multidimensional Random Subset Sum Problem

TL;DR

This paper extends the random subset sum problem to multiple dimensions, proving that a polynomial number of samples suffice for universal approximation capabilities in neural network models.

Contribution

It establishes a multidimensional generalization of the subset sum problem with bounds on sample size for approximation, and demonstrates neural network universality.

Findings

Multidimensional subset sum approximation with high probability.

Sample complexity bound of O(d^3 log(1/ε) (log(1/ε)+log d)).

Neural network universality with polynomial overhead.

Abstract

In the Random Subset Sum Problem, given $n$ i.i.d. random variables $X_1, ..., X_n$, we wish to approximate any point $z \in [-1,1]$ as the sum of a suitable subset $X_{i_1(z)}, ..., X_{i_s(z)}$ of them, up to error $\varepsilon$. Despite its simple statement, this problem is of fundamental interest to both theoretical computer science and statistical mechanics. More recently, it gained renewed attention for its implications in the theory of Artificial Neural Networks. An obvious multidimensional generalisation of the problem is to consider $n$ i.i.d. $d$-dimensional random vectors, with the objective of approximating every point $\mathbf{z} \in [-1,1]^d$. In 1998, G. S. Lueker showed that, in the one-dimensional setting, $n=\mathcal{O}(\log \frac 1\varepsilon)$ samples guarantee the approximation property with high probability.In this work, we prove that, in $d$ dimensions, $n = \mathcal{O}(d^3\log \frac 1\varepsilon \cdot (\log \frac 1\varepsilon + \log d))$ samples suffice for the approximation property to hold with high probability. As an application highlighting the potential interest of this result, we prove that a recently proposed neural network model exhibits universality: with high probability, the model can approximate any neural network within a polynomial overhead in the number of parameters.