The Oracle of DLphi

TL;DR

This paper introduces a deep learning and set theory-based method that achieves high accuracy in classification and prediction, even with unrelated training and test data, given sufficient labeled data.

Contribution

It presents a novel technique combining deep learning and set theory that can predict test labels accurately regardless of training data relevance, given enough data.

Findings

High prediction accuracy with unrelated training data

Method effective with large labeled datasets

Data relevance is not critical with sufficient data

Abstract

We present a novel technique based on deep learning and set theory which yields exceptional classification and prediction results. Having access to a sufficiently large amount of labelled training data, our methodology is capable of predicting the labels of the test data almost always even if the training data is entirely unrelated to the test data. In other words, we prove in a specific setting that as long as one has access to enough data points, the quality of the data is irrelevant.

Equations

$$\Phi (x)=T_L(\varrho (T_{L-1}(\dots \varrho (T_1(x))))),\text{ for all }x\in [0,1{]}^d,$$

$${\mathcal{NN}}_{d,\varrho }:=\bigcup _{L\in \mathbb{N}}{\mathcal{NN}}_{d,\varrho ,L}.$$

$$L\left(X,{\left(y_i^X\right)}_{i\in [0,1{]}^d}\right)={\left(y_i\right)}_{i\in [0,1{]}^d},$$

$$\mathcal{L}\left(X,{\left(y_i^X\right)}_{i\in [0,1{]}^d}\right)(j)=y_j,\text{ for all }j\in X.$$

$$f:[0,1{]}^d\to [0,1]:j\mapsto y_j$$

$${\int }_{[0,1{]}^d}\left|\mathcal{L}\left(X,(y_i^X{)}_{i\in [0,1{]}^d}\right)(j)-f(j)\right|dj<ϵ.$$

$$L:\mathcal{P}\left([0,1{]}^d\right)\times {[0,1]}^{([0,1{]}^d)}$$

$$L(X,(y_i{)}_{i\in [0,1{]}^d})(j)$$

$$\lambda \left([[(y_i{)}_{i\in [0,1{]}^d}]]\right)\sim (y_i{)}_{i\in [0,1{]}^d}$$

$$g:[0,1{]}^d\to [0,1]:i\mapsto z_i.$$

$${\int }_{[0,1{]}^d}|g(j)-{\Phi }_{ϵ}(j)|dj<\frac{ϵ}{2}.$$

$${\Phi }_{k,n}:[0,1{]}^d$$

$$j$$

$${\int }_{[0,1{]}^d}|{\Phi }_{k,n}(j)|dj<\frac{ϵ}{2|X|}\text{ and }\mathrm{supp}{\Phi }_{k,n}\cap \mathrm{supp}{\Phi }_{k^{\prime },n}=\varnothing \text{ for }k\ne k^{\prime }.$$

$${\Phi }_{ϵ,X}:={\Phi }_{ϵ}+\sum _{k\in X}{\Phi }_{k,n^{\ast }}.$$

$$\mathcal{L}(X,(y_i{)}_{i\in [0,1{]}^d}):={\Phi }_{ϵ,X}.$$

$$\mathcal{L}\left(X,{\left(y_i^X\right)}_{i\in [0,1{]}^d}\right)(k)={\Phi }_{ϵ}(k)+{\Phi }_{k,n^{\ast }}(k)=\lambda {\left({\left(y_i^X\right)}_{i\in [0,1{]}^d}\right)}_k$$

$${\int }_{[0,1{]}^d}\left|\mathcal{L}\left(X,{\left(y_i^X\right)}_{i\in [0,1{]}^d}\right)(j)-f(j)\right|dj$$

Full text

1 Introduction

This paper takes aim at achieving nothing less than the impossible. To be more precise, we seek to predict labels of unknown data from entirely uncorrelated labelled training data. This will be accomplished by an application of an algorithm based on deep learning, as well as, by invoking one of the most fundamental concepts of set theory.

Estimating the behaviour of a system in unknown situations is one of the central problems of humanity. Indeed, we are constantly trying to produce predictions for future events to be able to prepare ourselves. For example, the benefits of accurate weather forecast include allowing us to make the right choice of clothing for the day, deciding if we should go by bike or take the bus to work, or if we should seek shelter from a natural catastrophe [4]. Election predictions give companies the possibility of funding the most promising candidates and gain in influence. Analysing the history of class conflicts allows making reliable predictions of our future economic systems [27]. Finally, precise knowledge of the behaviour of the stock market enables banks to make profits and strategically destabilise the economy [10].

One of the most promising and successful techniques for such predictions are data-driven methods, most importantly deep learning (DL). Nonetheless, it has been conventional wisdom that not even deep learning techniques can predict labels of unseen data points if they have nothing to do with the training data. In this work, we challenge this paradigm and show that there is no need to use training data that is particularly correlated with the data points that should be predicted. Indeed, we will see below, that as long as one has a sufficient amount of training data, one can accurately predict everything, even events that are entirely independent of every data point used for training! In other words, not the quality of the training data is important but its sheer amount.

1.1 Related work

We will recall a couple of relevant articles that highlight the current efficiency of deep neural networks and then cite a number of our articles for no apparent reason. Neural networks were initially introduced in the 1940s by McCulloch and Pitts [28] in an attempt to mathematically model the human brain. Later, this framework was identified as a flexible and powerful computational architecture which then led to the field of deep learning [12, 23, 38]. Deep learning, roughly speaking, deals with the data-driven manipulation of neural networks. These methods turned out to be highly efficient to the extent that deep learning based methods are state-of-the-art technology in all image classification tasks [20, 41, 22]. They have revolutionised the field of speech recognition [16, 7, 47] and have achieved a level of skill in playing games that humans or the best alternative algorithms cannot match anymore [40, 43, 48].

Many mathematicians have since been trying to understand why these algorithms are superior to the classical approaches. It was observed that deep networks naturally produce invariances, which might help to explain their excellent behaviour as a high-dimensional classifier [5, 46]. The learning procedure is very unlikely to end up in bad local minima as has been observed in [44, 21]. Alternative approaches focus on approximation theory; we mention [6, 19, 2, 29, 30, 34, 39, 49] which is undoubtedly the second most biased selection of references we are about to make in this article.

A couple of additional papers that the reader should be aware of are [1, 13, 34, 9, 8, 33, 25, 35, 37, 32, 14, 11, 17, 3, 26, 45, 31]. If not for their relevance to deep learning, these articles at least give the reader an impression of the general interests of the authors.

1.2 Notation

For a set $\Omega$, we denote by $\mathcal{P}(\Omega)$ the power set of $\Omega$. In the following we will restrict out attention to the space $[0,1]^{d}$ for $d\in\mathbb{N}$ and unless specified differently we equip it with the Lebesgue measure, which in this case is a probability measure. For $1\leq p<\infty$, we denote by $L^{p}([0,1]^{d})$ the classical Lebesgue spaces. Finally, we denote by ${[0,1]}^{([0,1]^{d})}$ the set of all indexed sets $(y_{i})_{i\in[0,1]^{d}}$ where $y_{i}\in[0,1]$ or equivalently the set of all maps from $[0,1]^{d}$ to $[0,1]$.

2 Neural Networks

We start by introducing neural networks as functions that can be written as an alternating application of affine linear maps and an activation function.

Definition 2.1**.**

For $d,L\in\mathbb{N}$, a neural network with activation function $\varrho:\mathbb{R}\to\mathbb{R}$ is a function $\Phi:[0,1]^{d}\to\mathbb{R}$ such that there exist $N_{0},\dots,N_{L}\in\mathbb{N}$, where $N_{0}\coloneqq d$ and $N_{L}\coloneqq 1$ as well as affine linear maps $T_{\ell}:\mathbb{R}^{N_{\ell-1}}\to\mathbb{R}^{N_{\ell}}$ such that

[TABLE]

where $\varrho$ is applied coordinate-wise. The set of all neural networks which have a representation as above is denoted by $\mathcal{NN}_{d,\varrho,L}$. Additionally, we denote

[TABLE]

We recall one of the main results in neural network theory: the universal approximation theorem. The original statements are due to Hornik [18] and Cybenko [6], but we instead state a more general version describing approximation with respect to $L^{p}$ norms.

Theorem 2.2** ([24]).**

Let $d\in\mathbb{N}$, $\varrho:\mathbb{R}\to\mathbb{R}$ be continuous and assume that $\varrho$ is not a polynomial. Then, for all $L\geq 2$ and all $1\leq p<\infty$, $\mathcal{NN}_{d,\varrho,L}$ is dense in $L^{p}([0,1]^{d})$. In particular, $\mathcal{NN}_{d,\varrho}$ is dense in $L^{p}([0,1]^{d})$, for $1\leq p<\infty$.

3 Main Results

As already announced in the introduction, we aim to construct a prediction algorithm that can produce a correct classification of unseen data based on uncorrelated training data. This formulation is sufficiently vague to be considerably impressive; however, it lacks the necessary rigour to be considered a mathematical statement. To make the statement more precise, we introduce the following definitions.

Definition 3.1**.**

Let $\mu$ be the Lebesgue measure on $[0,1]^{d}$ (which corresponds to the uniform probability distribution on $[0,1]^{d}$) and let $\nu$ be any discrete probability distribution on $\mathbb{N}$. We call a set $X$ obtained as the union of $k\in\mathbb{N}$ samples $X_{1},\dots,X_{k}$ drawn i.i.d. according to $\mu$ a random set of size at most $k$. We call a set $X$ generated by the following sampling procedure a random set of finite size (or simply a finite random set): First draw $k\in\mathbb{N}$ according to $\nu$, then generate a random set of size at most $k$ as before.

Note that for the uniform distribution $\mu$ we will almost surely generate a set of size exactly $k$ (as opposed to at most $k$) by drawing $k$ i.i.d. samples for every $k\in\mathbb{N}$. It is not hard to see that for any finite $S\subset[0,1]^{d}$ and any random finite set $X\subset[0,1]^{d}$ the probability of $S$ and $X$ having non-empty intersection is zero. This is independent of $\nu$ as long as the uniform measure $\mu$ is used for the sampling.

Definition 3.2**.**

Let $d\in\mathbb{N}$. A map $L:\mathcal{P}([0,1]^{d})\times{[0,1]}^{([0,1]^{d})}\to{[0,1]}^{([0,1]^{d})}$ is said to possess the precise prediction property if for every set of labelled data $(y_{i})_{i\in[0,1]^{d}}\in{[0,1]}^{([0,1]^{d})}$ it holds that, for almost all finite random sets $X\subset[0,1]^{d}$,

[TABLE]

where $y_{i}^{X}\coloneqq y_{i}$ if $i\not\in X$ and $y_{i}^{X}\coloneqq 0$ if $i\in X$.

An honest description of Definition 3.2 is that given a ground truth $(y_{i})_{i\in[0,1]^{d}}$ of labelled data, possibly without any inherent structure, there exists a learning algorithm that, when presented with a large subset of the labelled data (namely with $(y_{i})_{i\in[0,1]^{d}\setminus X}$ for a generic finite $X\subset[0,1]^{d}$), can predict the missing data almost surely.

It turns out that there is an algorithm that, from the masked data $(y_{i}^{X})_{i\in[0,1]^{d}}$, produces a deep neural network predicting the labels $(y_{i})_{i\in X}$ almost surely. This deep network can be the basis of a map that possesses the precise prediction property. We shall present this result in the following theorem. What is more is that the proof below is semi-constructive. All that is required to perform the construction of a the deep neural network is an unwavering belief in the axioms of set theory, a potentially significant amount of book-keeping, and the solution of a moderate regression problem. Naturally, conventional wisdom suggests that the explicit construction of the proof can be replaced by training a sufficiently large and deep neural network via stochastic gradient descent.

Theorem 3.3**.**

Let $\varrho:\mathbb{R}\to\mathbb{R}:$ $\varrho(x)\coloneqq\max\{0,x\}$. Then, for all $\epsilon>0$, there exists a map $\mathcal{L}:\mathcal{P}([0,1]^{d})\times{[0,1]}^{([0,1]^{d})}\to\mathcal{NN}_{d,\varrho}$ such that: For every set of labelled data $(y_{i})_{i\in[0,1]^{d}}\in{[0,1]}^{([0,1]^{d})}$ it holds that, for almost all finite random sets $X\subset[0,1]^{d}$,

[TABLE]

If, additionally, the map

[TABLE]

is integrable, then we have that

[TABLE]

In particular, the map

[TABLE]

has the precise prediction property.

Proof.

We define an equivalence relation $\sim$ on ${[0,1]}^{([0,1]^{d})}$ by defining $(y_{i})_{i\in[0,1]^{d}}\sim(z_{i})_{i\in[0,1]^{d}}$ if $y_{i}=z_{i}$ for all but finitely many $i\in[0,1]^{d}$. We denote the equivalence class containing $(y_{i})_{i\in[0,1]^{d}}$ by $[[(y_{i})_{i\in[0,1]^{d}}]]$. By the axiom of choice, there exists a function $\lambda$ that maps every equivalence class $\Xi\subset{[0,1]}^{([0,1]^{d})}$ to an associated representative $(y_{i})_{i\in[0,1]^{d}}\in\Xi$, i.e.,

[TABLE]

for all $(y_{i})_{i\in[0,1]^{d}}\in{[0,1]}^{([0,1]^{d})}$.

Now let $\epsilon>0$. Let $X\subset[0,1]^{d}$ (not necessarily finite) and let $(y_{i})_{i\in[0,1]^{d}}\in{[0,1]}^{([0,1]^{d})}$. Let $(z_{i})_{i\in[0,1]^{d}}\coloneqq\lambda([[(y_{i})_{i\in[0,1]^{d}}]])$ be the representative of the equivalence class $[[(y_{i})_{i\in[0,1]^{d}}]]$. We proceed by defining several neural networks and consider the case of infinite and finite $X$ separately.

First, in the case of infinite $X$ we simply set $\Phi_{\epsilon,X}=0$.

Second, in the case of finite $X$ let

[TABLE]

If $g$ is measurable and integrable then by Theorem 2.2 there exists a neural network $\Phi_{\epsilon}$ such that

[TABLE]

Otherwise, we set $\Phi_{\epsilon}=0$. Now further, for every $k\in X$ set $r_{k}\coloneqq g(k)-\Phi_{\epsilon}(k)$ and define

[TABLE]

where $j_{\ell}$ and $k_{\ell}$ denote the $\ell$-th coordinate of $j$ and $k$ respectively. It is not hard to see that $\Phi_{k,n}(k)=r_{k}$ and that there exists $n^{*}$ such that for all $n\geq n^{*}$

[TABLE]

Now define

[TABLE]

Finally, we define

[TABLE]

Clearly, $\mathcal{L}(X,(y_{i})_{i\in[0,1]^{d}})$ is a neural network. Moreover, by construction, if $X$ is finite and $k\in X$,

[TABLE]

and $\lambda\left(((y_{i}^{X})_{i\in[0,1]^{d}}\right)\sim(y_{i}^{X})_{i\in[0,1]^{d}}\sim(y_{i})_{i\in[0,1]^{d}}$. Therefore, the set on which $\lambda\left(y_{i}^{X})_{i\in[0,1]^{d}}\right)_{k}$ and $(y_{i})_{i\in[0,1]^{d}}$ differ is finite and thus the probability of it intersecting $X$ is zero if $X$ is a finite random set. This concludes the first part of the theorem.

For the second part observe that if $f$ is integrable, then since $g=f$ except on a finite set of measure zero, also $g$ is integrable and thus by the triangle inequality

[TABLE]

This completes the proof. ∎

4 Discussion

Theorem 3.3 implies that deep neural networks are, in principle, able to predict data, that is completely uncorrelated with the training data. This observation might appear unintuitive, stunning, and maybe unbelievable at first, but the same can be said for most other deep learning results. We shall address a number of limitations and observations about Theorem 3.3 below.

In a sense, the algorithm requires an uncountable number of data points. Such a situation is often excluded in applications despite the current trend of big data. It is, however, expected that the amount of data that we need to handle will increase with time. Following this reasoning, scholars have speculated that we might soon enter the era of very big data.

Many machine learning algorithms suffer from overfitting, i.e., their predictions are overly adapted to the training data, and they do not generalise to unseen data. Equation (3.1) shows that the neural network produced by $\mathcal{L}$ predicts the labels of the points in $X$ correctly with probability $1$. However, on the training set, the accuracy is a bit worse as demonstrated by Equation (3.2). In other words, $\mathcal{L}$ is not overfitting, but deliberately underfitting while the prediction remains accurate.

It has often been conjectured that deep learning techniques can remove the curse of dimensionality that plagues many high-dimensional approximation tasks. Careful inspection of the statement of Theorem 3.3 indeed reveals that the quality of prediction appears to be independent of the ambient dimension $d$. Another piece of conventional wisdom is that deep networks are more efficient than their shallow counterparts, [42, 36]. The construction of (3.3) requires at least two hidden layers. Thus, the network is technically not a shallow network. It is fair say that this observation gives yet another illustration of the power of depth.

Finally, we mention that the construction is inspired by a prediction strategy for ordered data, presented in [15]. The discussion in that paper indicates that the same result is most likely not valid in classical Zermelo-Fraenkel set theory. It might not be wise to prominently suggest that the current deep learning revolution and the overwhelming empirical success prove that the axiom of choice is an empirically reasonable assumption, but we leave the reader to her own conclusion/s.

Acknowledgements

All authors except for Philipp Grohs are indebted to Philipp Grohs for giving an especially lovely lecture at the Oberwolfach Seminar ”Deep Learning” in October of 2018. This lecture led the participants to profound insights and culminated in this article and hopefully many more. The authors are grateful for the hospitality of the Mathematisches Forschungsinstitut Oberwolfach which hosted this fantastic event.

Disclaimer

We concede that this manuscript has mostly humouristic value. However, while the main theorem is based on a dubious application of the axiom of choice, it is a correct mathematical statement. Therefore, this manuscript at least highlights the dangers of applying mathematical theory to real-world applications blindly.