Sparse mean localization by information theory

TL;DR

This paper explores a theoretical approach to sparse mean estimation using information theory, specifically generalization capacity, and proposes an importance sampling algorithm to handle large hypothesis spaces effectively.

Contribution

It introduces a novel application of generalization capacity to sparse mean estimation and develops an importance sampling algorithm for large hypothesis spaces.

Findings

Generalization capacity guides resolution selection in sparse mean estimation.

The importance sampling algorithm efficiently handles exponential hypothesis spaces.

The approach improves understanding of approximation set coding in high-dimensional settings.

Abstract

Sparse feature selection is necessary when we fit statistical models, we have access to a large group of features, don't know which are relevant, but assume that most are not. Alternatively, when the number of features is larger than the available data the model becomes over parametrized and the sparse feature selection task involves selecting the most informative variables for the model. When the model is a simple location model and the number of relevant features does not grow with the total number of features, sparse feature selection corresponds to sparse mean estimation. We deal with a simplified mean estimation problem consisting of an additive model with gaussian noise and mean that is in a restricted, finite hypothesis space. This restriction simplifies the mean estimation problem into a selection problem of combinatorial nature. Although the hypothesis space is finite, its size is exponential in the dimension of the mean. In limited data settings and when the size of the hypothesis space depends on the amount of data or on the dimension of the data, choosing an approximation set of hypotheses is a desirable approach. Choosing a set of hypotheses instead of a single one implies replacing the bias-variance trade off with a resolution-stability trade off. Generalization capacity provides a resolution selection criterion based on allowing the learning algorithm to communicate the largest amount of information in the data to the learner without error. In this work the theory of approximation set coding and generalization capacity is explored in order to understand this approach. We then apply the generalization capacity criterion to the simplified sparse mean estimation problem and detail an importance sampling algorithm which at once solves the difficulty posed by large hypothesis spaces and the slow convergence of uniform sampling algorithms.

Equations

$$X_i={\mu }^0+{ϵ}_i$$

$$|\mathcal{C}(O^{(n)})|<\infty$$

$$c^{\ast }(X^{(n)})$$

$$\hat{R}(c,X^{(n)})$$

$$w:\mathcal{C}\times {\mathcal{X}}^n\times {\mathbb{R}}^{+}\to [0,1]$$

$$(c,X^{(n)},\beta )\mapsto w_{\beta }(c,X^{(n)})$$

$$w_{\beta }(c,X^{(n)})=\{\begin{array}{ll}1,& \hat{R}(c,X^{(n)})\le \hat{R}(c^{\ast },X^{(n)})+\text{nicefrac}1\beta \\ 0,& \text{otherwise}\end{array}$$

$$w_{\beta }(c,X^{(n)})\ge w_{\beta }(c^{\prime },X^{(n)})\iff \hat{R}(c,X^{(n)})\le \hat{R}(c^{\prime },X^{(n)})$$

$$w_{\beta }^f(c,X^{(n)})=\{w_{\beta }(c,X^{(n)})=e^{-\beta f(\hat{R}(c,X^{(n)}))}:\beta \in {\mathbb{R}}^{+},f\text{ increasing}\}$$

$$w_{\beta }(c,X^{(n)}):=e^{-\beta \hat{R}(c,X^{(n)})}$$

$$Z_{\beta }(X^{(n)}):=\sum _{c\in \mathcal{C}(O^{(n)})}{w}_{\beta }(c,X^{(n)})$$

$$P_G(c;\beta ,X^{(n)}):=\frac{w_{\beta }(c,X^{(n)})}{Z_{\beta }(X^{(n)})}$$

$${\mathbb{E}}_{p(c)}[\hat{R}(c,X^{(n)})]={\mu }_{\beta }$$

$$\Delta Z_{\beta }(X^{\prime },X^{\prime \prime })=\sum _{c\in \mathcal{C}}{w}_{\beta }(c,X^{\prime })w_{\beta }(c,X^{\prime \prime })$$

$$I_{\beta }:=\log \frac{|\mathcal{C}|\Delta Z_{\beta }(X^{\prime },X^{\prime \prime })}{Z_{\beta }(X^{\prime })Z_{\beta }(X^{\prime \prime })}$$

$$0$$

$$=\frac{{\sum }_{i=1}^{|\mathcal{C}|}{a}_i{b}_i}{{\sum }_{i=1}^{|\mathcal{C}|}{\sum }_{j=1}^{|\mathcal{C}|}{a}_i{b}_j}=\frac{{\sum }_{i=1}^{|\mathcal{C}|}{a}_i{b}_i}{{\sum }_{i=1}^{|\mathcal{C}|}{a}_i{b}_i+{\sum }_{i\ne j}{a}_i{b}_j}\le 1$$

$$I:=\underset{\beta \in {\mathbb{R}}^{+}}{\max }{\mathbb{E}}_{(X^{\prime },X^{\prime \prime })}{I}_{\beta }$$

$$I(x)=-\log p(x)$$

$$H(X):=-{\mathbb{E}}_X[\log p(X)]={\mathbb{E}}_X[I(X)]=\sum _{x\in \mathcal{X}}p(x)\log p(x)$$

$$H(\mathcal{W}):=-\sum _{i=1}^M\frac{1}{M}\log \frac{1}{M}=\log M$$

$$H(X,Y):=-\sum _{(x,y)\in \mathcal{X}\times \mathcal{Y}}p(x,y)\log p(x,y)=-{\mathbb{E}}_{(X,Y)}\log p(X,Y)$$

$$H(Y|X):=-\sum _{(x,y)\in \mathcal{X}\times \mathcal{Y}}p(x,y)\log p(y|x)=-{\mathbb{E}}_{(X,Y)}\log p(Y|X)$$

$$I(X;Y):=\sum _{(x,y)\in \mathcal{X}\times \mathcal{Y}}p(x,y)\log \frac{p(x,y)}{p(x)p(y)}={\mathbb{E}}_{(X,Y)}\log \frac{p(X,Y)}{p(X)p(Y)}$$

$$Y_t\perp \perp \{X_1,Y_1,X_2,Y_2,...,X_{t-1},Y_{t-1}\}|X_t$$

$$C:=\underset{p(x)}{\max }I(X;Y)$$

$$p(x^{\prime \prime }|x^{\prime })\propto p(x^{\prime },x^{\prime \prime })=\sum _{c\in \mathcal{C}}p(c,x^{\prime },x^{\prime \prime })$$

$$=\frac{1}{M}\sum _{c\in \mathcal{C}}p(x^{\prime }|c)p(x^{\prime \prime }|c)$$

$${\mathcal{T}}^h=\{t^h\ne u^h\in \mathcal{T}:\forall c\ne d\in \mathcal{C}\Rightarrow t^h(c)\ne t^h(d)\text{ and }{t}^h(c)\ne u^h(c)\}$$

$${\mathcal{T}}^D=\{t^D:{\mathcal{X}}^n\to {\mathcal{X}}^n\}$$

$$p(c|x)=p(t^h(c)|t^D(x))$$

$$p(c|x)\propto p(x|c)p(c)$$

Full text

Sparse Mean Localization

by Information Theory

I would like to express my gratitude to Stefan Bauer and Nico Gorbach who gave me all the support I needed, to Prof. Dr. Joachim Buhmann for the opportunity to work in such as a fascinating topic, and to Dr. Markus Kalisch for all the support, in this semester project and throughout the Masters’ programme.

Abstract

Sparse feature selection is necessary when we fit statistical models, we have access to a large group of features, don’t know which are relevant, but assume that most are not. Alternatively, when the number of features is larger than the available data the model becomes overparametrized and the sparse feature selection task involves selecting the most informative variables for the model. When the model is a simple location model and the number of relevant features does not grow with the total number of features, sparse feature selection corresponds to sparse mean estimation. We deal with a simplified mean estimation problem consisting of an additive model with gaussian noise and mean that is in a restricted, finite hypothesis space (parameter space). This restriction simplifies the mean estimation problem into a selection problem of combinatorial nature. Although the hypothesis space is finite, its size is exponential in the dimension of the mean. In limited data settings and when the size of the hypothesis space depends on the amount of data or on the dimension of the data, choosing an approximation set of hypotheses is a desirable approach. Choosing a set of hypotheses instead of a single one implies replacing the bias-variance trade off with a resolution-stability trade off. Generalization capacity provides a resolution selection criterion based on allowing the learning algorithm to communicate the largest amount of information in the data to the learner without error. In this work the theory of approximation set coding and generalization capacity is explored in order to understand this approach. We then apply the generalization capacity criterion to the simplified sparse mean estimation problem and detail an importance sampling algorithm which at once solves the difficulty posed by large hypothesis spaces and the slow convergence of uniform sampling algorithms (caused by the skewed distribution of hypothesis costs). Finally we explore how the generalization capacity criterion can be a applied to a more realistic version of the sparse feature selection problem where the number of relevant features grows with the total number of features.

Contents

@starttoc

toc

List of Figures

@starttoc

lof

Chapter 1 Introduction

It is often the case that when fitting statistical models, the majority of available features are not informative in the sense of the underlying learning task. In other cases the limited amount of data available implies that most features can’t be used, even if they are all informative, because the model becomes overparametrized. In both instances sparse feature selection must be done prior or simultaneous to model fitting. In this work we deal with the sparse feature selection problem as it applies to a simplified location model. We first assume the number of relevant features is small and fixed and then explore the case where the number of relevant features is small but grows with the total number of features. Although this problem is well known and studied, for example in vanDeGeer, we are interested in how we can apply approximation set coding and generalization capacity to localize the hypothesis class to an optimal resolution.

1.1 Structure

The report is organized as follows. Section 1.2 gives a description of the problem we will focus on: sparse mean estimation and sparse feature selection. We want to solve this problem using the approximation set coding and generalization capacity methodology proposed by Buhm13, so in Sections 1.3-1.6 we give an introduction to the theory involved. Section 1.3 introduces the pattern analysis framework for learning problems. In section 1.4 we explore how, by defining approximation sets of hypotheses instead of proposing a single hypothesis as the solution, we are able to move from the normal bias-variance trade-off of learning problems to a resolution-stability trade-off. In section 1.5, with the help of concepts from Sections 1.3 and 1.4, we define various information theoretic concepts such as Boltzmann weights, Gibbs distributions and partition functions, culminating in the definition of generalization capacity. We try to give an intuitive understanding of each of these concepts except that of generalization capacity itself. In Section 1.6 we motivate the concept of generalization capacity in analogy to Shannon’s noisy channel coding theorem from which it is derived.

In Chapter 2 we estimate the generalization capacity of the squared loss based, empirical risk function for the non-sparse version of the mean localization problem. We concentrate on low-dimensional cases. In Section 2.1 the information theoretic concepts defined in Section 1.5 are applied to the problem at hand culminating in an expression for the generalization capacity that suggests an exhaustive simulating algorithm for its estimation. Section 2.2 includes the pseudo code for implementing this algorithm. Section 2.3 includes the results of implementing the exhaustive simulating algorithm to estimating generalization capacity. Section 2.4 is a note on how to avoid underflow problems when implementing this algorithm. In Section 2.5 we explore different ways in which we may incoprorate the use of common random numbers into our algorithm as a variance reduction technique.

In Chapter 3 we estimate the generalization capacity for the sparse mean localization problem. Section 3.1 describes the changes and additional tools necessary to implement the algorithm described in 2.2 to the sparse version of the problem. Section 3.2 includes the results of implementing this algorithm to estimating generalization capacity. Since the algorithm will be shown to be inadequate in the high dimensional case, in Section 3.3 we describe a sampling algorithm based on a re-expression of the generalization capacity. Section 3.4 includes the results of this sampling algorithm. This algorithm will be shown to converge too slowly in the number of simulations and so in section 3.5 we describe an importance sampling algorithm for estimating generalization capacity. Section 3.6 includes the results of this algorithm.

Chapter 4 is a brief exploration into a more realistic version of sparse feature selection where the number of relevant features grows with the total number of features. We describe the problem and explore some of the difficulties of estimating generalization capacity with a simulation algorithm in this case.

Chapter 5 includes a summary of the report and a list of possible related avenues of future research.

1.2 Problem statement

We deal with the statistical model studied in Buhm14:

[TABLE]

where

• •

$\mu^{0}\in\mathbb{B}^{d}=\{0,1\}^{d}$

• •

$X_{i},\epsilon_{i}\in\mathbb{R}^{d}$

• •

$\epsilon\sim N(0,\sigma^{2}I_{d})$

• •

observations $X_{i}$ with $i=1,...,n$ are i.i.d.

In the general case estimating $\mu^{0}$ corresponds to selecting a hypothesis $\mu$ from the hypothesis space $\mathbb{B}^{d}$ which has cardinality $2^{d}$. While we first deal with this problem, we will be more interested in a modified version of this problem where:

1. i.)

$||\mu^{0}||_{1}=k$ 2. ii.)

$\mu^{0}\in\mathbb{B}^{d}_{k}=\{\mu\in\mathbb{B}^{d}:||\mu||_{1}=k\}$ 3. iii.)

$|\mathbb{B}^{d}_{k}|={d\choose k}$ 4. iv.)

It is assumed that $k$ is known.

We first deal with the general case where $k,d\in\mathbb{Z}^{+}$, $k\leq d$, and then with a sparse case where $k$ is kept constant and $d$ grows toward infinity. In Chapter 4 we briefly discuss another sparsity condition where $k\approx\frac{d}{\log(d)}$.

1.3 Pattern analysis

Although the classical framework of parameter inference, in which estimators $\hat{\theta}(\cdot)$ are maps from a sample space $\mathcal{X}$ to a parameter space $\Theta$, is appropriate for the problem at hand we introduce Aproximation Set Coding (ASC) and Generalization Capacity (GC) within the framework of Pattern Analysis since they are more relevant in this wider context. The rest of this introductory chapter follows Buhm13 closely.

The problem described in Section 1.2 belongs to the class of problems which are the object of Pattern Analysis. The goal of pattern analysis is to map a set of object configurations to a pattern space. Concretely, we want to choose a hypothesis $c\in\mathcal{C}(O^{(n)})$ where:

• •

$O_{i}\in\mathcal{O}$ are objects in an object space.

• •

$O^{(n)}=\{O_{1},...,O_{n}\}\in\mathcal{O}^{(n)}$ are object sets.

• •

$c:\mathcal{O}^{(n)}\rightarrow\mathcal{P}$ is a hypothesis in a hypothesis class $\mathcal{C}(O^{(n)})$ and $\mathcal{P}$ is a pattern space.

A few remarks about this framework:

1. i.)

The hypothesis class $\mathcal{C}(O^{(n)})$ may or may not depend on the object set. Specifically, the size of the hypothesis class may depend on the object set or not. 2. ii.)

In this exposition the objects in the object set $O_{i}\in\mathcal{O}$ may be tuples of objects from more fundamental object sets, i.e. $O_{i}=(o_{i1},...,o_{ir}),o_{ij}\in\mathcal{O}_{j}$. However, the objects $O_{i}$ are at the level of the mapping $c$. 3. iii.)

The hypothesis map $\mathcal{C}(O^{(n)})$ is actually a composition of the maps $X:\mathcal{O}^{(n)}\rightarrow\mathcal{X}^{n}$ and $t:\mathcal{X}^{n}\rightarrow\mathcal{P}$ where $\mathcal{X}^{n}$ is a measurement space. 4. iv.)

The pattern space $\mathcal{P}$ may be related to the data generating process or not. It is an interpretation space: a set of abstract, mutually exclusive properties which we wish to assign to object configurations.

We present some examples to clarify the pattern analysis framework.

Example 1.3.0.1** (Mean estimation).**

We want to estimate the population mean height of swiss women given a sample of 100. We make no assumptions regarding the data generating process.

• •

$\mathcal{O}=\{$swiss women$\}$**

• •

$O^{(100)}$* is the set of sampled women.*

• •

$X(O^{(100)})$* are the heights of the sampled women.*

• •

$\mathcal{X}^{n}\subset(\mathbb{R}^{+})^{100}$* is the set of possible heights for the 100 women.*

• •

$\mathcal{P}\subset\mathbb{R}^{+}$* is the set of possible population mean heights.*

• •

$\mathcal{C}(O^{(100)})=\mathcal{O}^{(100)}\times\mathcal{P}$* is the hypothesis class which does not depend on the size $n=100$ of the object set.*

Example 1.3.0.2** (Clustering - Population).**

We want to cluster 100 people into 4 groups according to height and weight. We assume the underlying data generating process is a gaussian mixture with parameters $\{(\mu_{1},\Sigma_{1}),...,(\mu_{4},\Sigma_{4})\}$ with $\mu_{i}\in\mathbb{R}^{2}$ and $\Sigma_{i}\in\mathbb{R}^{2\times 2}$.

• •

$\mathcal{O}=\{$people$\}$.

• •

$O^{(100)}$* is the set of sampled people.*

• •

$X(O^{(100)})=\{(h_{1},w_{1}),...,(h_{100},w_{100})\}$* are the heights and weights of the sampled people.*

• •

$\mathcal{X}^{n}\subset\mathbb{R}^{100\times 2}$* is the set of possible heights and weights for the 100 people.*

• •

$\mathcal{P}\subset\mathbb{R}^{2\times 4}\times\mathbb{R}^{2\times 2\times 4}$* is the set of possible population mean and covariances.*

• •

$\mathcal{C}(O^{(100)})=\mathcal{O}^{(100)}\times\mathcal{P}$* is the hypothesis class which does not depend on the size $n$ of the object set.*

Remark: Notice how our assumptions about the data generating process inform our choice of pattern space $\mathcal{P}$.

Example 1.3.0.3** (Clustering - Sample).**

We want to cluster 100 people into 4 groups according to height and weight. We do not assume anything about the underlying data generating process and are just interested in finding a clustering that defines homogenous groups for this sample and not the entire population.

• •

$\mathcal{O}=\{$people$\}$.

• •

$O^{(100)}$* is the set of sampled people.*

• •

$X(O^{(100)})=\{(h_{1},w_{1}),...,(h_{100},w_{100})\}$* are the heights and weights of the sampled people.*

• •

$\mathcal{X}^{n}\subset\mathbb{R}^{100\times 2}$* is the set of possible heights and weights for the 100 people.*

• •

$\mathcal{P}=\{1,2,3,4\}^{100}$* are all the possible ways we can group 100 people into 4 groups.*

• •

$\mathcal{C}(O^{(100)})=\mathcal{O}^{(100)}\times\mathcal{P}$* is the hypothesis class which in this case does depend on the size $n$ of the object set.*

Example 1.3.0.4** (Dyadic data).**

We are interested in predicting if a user will make a purchase at a given website, based on the age and gender of the person and on the type of website (there are $m$ types). We don’t assume anything about the data generating process but have already decided to model the probability of purchase using a logistic regression model.

• •

$\mathcal{O}=\mathcal{O}_{p}\times\mathcal{O}_{w}=\{$people$\}\times\{$websites$\}$.

• •

$O^{(n)}=\{(o_{1p},o_{1w}),...,(o_{np},o_{nw})\}$* is the set of sampled person-website pairs.*

• •

$X(O^{(n)})=\{(a_{1},g_{1},t_{1},p_{1}),...,(a_{n},g_{n},t_{n},p_{n})\}$* are the age, geneder, website-type and purchase outcome of the sampled person-website pairs.*

• •

$\mathcal{X}^{n}\subset\mathbb{R}^{+}\times\{male,female\}\times\{1,...,m\}\times\{0,1\}$* is the sample space.*

• •

$\mathcal{P}=\{(\beta_{0},\beta_{1},\beta_{2})\in\mathbb{R}^{3}\}$* is the parameter space for the logistic model.*

• •

$\mathcal{C}(O^{(n)})=\mathcal{O}^{(n)}\times\mathcal{P}$* is the hypothesis class which does not depend on the size $n$ of the object set.*

As we can see the pattern analysis framework fits a wide range of problems. Although problems such as mean estimation and regression, in which hypothesis classes with infinite cardinality are involved, can be tackled using the pattern analysis framework, in the rest of this introductory chapter we focus on classes with a finite number of hypotheses. In other words we assume:

[TABLE]

1.4 Approximation sets

In classical statistical learning theory, in order to solve an inference decision problem, we choose a loss function $\rho(c,x)$ with which we construct the risk function $R(c)=\mathbb{E}_{X}[\rho(c,X)]$. We then choose a single hypothesis $c^{*}$ that minimizes the empirical risk $\hat{R}(c,X^{(n)})$ for a given data set $X^{(n)}$:

[TABLE]

If $n$ is large then the Emprirical Risk Minimizer (ERM) will be close to the minimizer of the risk function $R(c,X^{(n)})$. However, in general we know that when $n$ is not large then the ERM will tend to overfit the data. Instead of choosing a single hypothesis we can choose a subset of the hypothesis class which includes good hypotheses: hypothesis with low costs. Qualitatively, we would like this set to be composed of low cost hypotheses which we cannnot (partially) order further because their costs are statistically indistinguishable. The goal is to choose a subset of hypotheses that are stable with respect to fluctuations in the cost measurements. We may code this selection with a weight function $w_{\beta}(c,X^{(n)})$ over the hypothesis class where:

[TABLE]

[TABLE]

[TABLE]

Where $\beta$ can be interpreted as the degree of certainty we have that $c^{*}$ is the best solution. We may generalize the concept of approximation sets by allowing fuzzy, non-binary selection where hypotheses belong to the solution set to varying degrees, i.e. $w_{\beta}(c,X^{(n)})\in[0,1]$. In this case valid weight vectors satisfy:

[TABLE]

The sum of the weights over the hypothesis class indicates the equivalent number of hypotheses selected. The bigger this sum the more unsure we are about $c^{*}$. We will sometimes say that $w_{\beta}(c,X^{(n)})$ is the approximation set of hypotheses, meaning that it encodes the (fuzzy) membership of the hypotheses in the set. A parametric family of weights which satisfies condition 1.4.0.6 is:

[TABLE]

Notice that if we normalize the weights such that $\sum_{c\in\mathcal{C}(O^{(n)})}w_{\beta}(c,X^{(n)})=1$ we can interpret the weights as a posterior probability distribution over the hypothesis class.

In the classical statistical setting, when we have limited data, obtaining unbiased estimators often means these estimators have high variance: estimations change dramatically from one data set to the next. Lowering the variance can sometimes be achieved by introducing bias into our estimator. This is the bias-variance trade-off that, when there is limited data, is usually resolved through some sort of regularization. As we shall see in Section 1.6 the ASC approach leads to a resolution-stability trade-off which replaces the bias-variance trade-off. Resolution refers to the equivalent number of hypotheses selected while stabiity refers to obtaining similar approximation sets for different $X^{(n)}\in\mathcal{X}^{n}$. Adopting the ASC approach the trade-off becomes, do we obtain a very stable set of good hypotheses that don’t change a lot depending on the data set but that is quite large (low resolution) or do we focus in on a small number of very good hypotheses but such that they will change from one data set to the next (unstable).

Notice that the $\beta$ parameter in our weight function $w_{\beta}$ is the resolution parameter that determines how this trade-off is resolved. Adopting the view of our normalized weight vector as a posterior over the hypothesis class, the higher $\beta$ is the more probability is spread or smoothed among all the hypotheses. In limited data settings, choosing the parameter $\beta$ corresponds to regularizing our empirical risk function.

In general, the justification for using the ASC approach is:

1. I

Inference. It allows us to identify hypotheses which are similar in cost but which might be distinct according to other criteria not included in the cost function. This benefit is also common to bayesian inference. 2. II

Learnability. For $\mathcal{C}(O^{(n)})$ to be learnable, ERM theory requires that it should not be too complex. In other words, $\mathcal{C}(O^{(n)})$ should have a finite VC-dimension. For certain problems, such as 1.3.0.3, the size of $\mathcal{C}(O^{(n)})$ increases too quickly in $n$, meaning that as $n\rightarrow\infty$ the empirical risk minimizer does not converge to the true risk minimizer. For this type of problem it is not even theoretically possible to converge to the true $c\in\mathcal{C}(O^{(n)})$ as $n\rightarrow\infty$ so an approximation set solution seems more reasonable.

1.5 Generalization capacity

Definition 1.5.0.1** (Boltzmann weights).**

If, from the parametric family 1.4.0.7, we choose $f(x)=x$ to construct our weight vector we obtain the so called Boltzmann weights:

[TABLE]

Definition 1.5.0.2** (Partition function).**

The sum over $\mathcal{C}(O^{(n)})$ of the Boltzmann weights is a function of the data $X^{(n)}$. We call it the partition function with respect to $w_{\beta}$ and define it as:

[TABLE]

Definition 1.5.0.3** (Gibbs Distribution).**

The normalized Boltzmann weights define a Gibbs distribution, $P_{G}(c;\beta,X^{(n)})$ over $\mathcal{C}(O^{(n)})$ with respect to the cost function $\hat{R}(c,X^{(n)})$:

[TABLE]

This choice of weight vector can be justified from an information theoretic perspective. The Gibbs distribution is the maximum entropy distribution among all distributions $p(c)$ over $\mathcal{C}(O^{(n)})$ such that:

[TABLE]

where $\mu_{\beta}$ is a non-increasing function of $\beta$. As we increase $\beta$, the resolution parameter, the expected cost with respect to the Gibbs distribution, decreases. In the limit, as $\beta\rightarrow\infty$, the Gibbs distribution becomes a single point mass distribution over $c^{*}$ and $\mu_{\beta}\rightarrow\hat{R}(c^{*},X^{(n)})$. The Gibbs distribution $P_{G}(c;\beta,X^{(n)})$ preserves the same (partial) ordering of $\mathcal{C}(O^{(n)})$ as $-\hat{R}(c,X^{(n)})$, but rescales so that differences in cost on the low end of the cost spectrum are exaggerated and differences in cost on the high end of the cost spectrum are smoothed out.

Figure 1.1 illustrates the mechanics of the smoothing of costs with the Boltzmann weight function. Costs scaled by the $\beta$ parameter are plotted on the x-axis ($x=\beta\hat{R}(c,X^{(n)})$) and the corresponding Boltzmann weights on the y-axis ($y=w_{\beta}(c,X^{(n)})=e^{-x}$). The black lines show the Boltzmann weights for two different resolution values: $\beta=20,1.5$. The green line represents the points $(\nicefrac{{\log(\beta)}}{{\beta}},\nicefrac{{1}}{{\beta}})$ which are the points that satisfy $\frac{dy}{dx}=-1$. Let the point of intersection between a given weight function $w_{\beta}(c,X^{(n)})$ indexed by $\beta$ and the green line be called the critical point $(x_{\beta}^{c},y_{\beta}^{c})$ for that $\beta$. For a given $\beta$, scaled costs to the right of the critical point $x_{\beta}^{c}$ (high costs) are smoothed onto the interval $(0,y_{\beta}^{c})$ while scaled costs to the left of $x_{\beta}^{c}$ (low costs) are exaggerated onto the interval $(y_{\beta}^{c},1)$. This is how the Boltzmann weight function $w_{\beta}(c,X^{(n)})$ and the parameter $\beta$ control the level of smoothing: for high resolution $\beta$ only the lowest cost hypotheses remain relevant, while for low resolution levels most hypotheses retain some measure of relevance.

The characteristics discussed above, are shared by all functions from the parametric family 1.4.0.7. These characteristics allow the Boltzmann weight function to be used in global optimization strategies such as simulated and deterministic annealing where the smoothing out of less important features in the cost surface in early iterations prevent the search algorithm from getting stuck in local minima. As is established in Jaynes1, Jaynes2 and Jaynes3, particular to the Gibbs distribution (for which $f(x)=x$), is the fact that for a given level of resolution, manifested as an expectation, $\mathbb{E}_{p(c)}[\hat{R}(c,X^{(n)})]=\mu_{\beta}$, that is a certain distance from $\hat{R}(c^{*},X^{(n)})$, it has maximum entropy among distributions with this characteristic This means that if we use the Gibbs distribution to describe our uncertainty about the true hypothesis, the only information extracted from $X^{(n)}$ is that obtained using $\hat{R}(c,X^{(n)})$. Interpreting the entropy of a distribution as a measure of its uncertainty and supposing we know that $\mathbb{E}_{p(c)}[\hat{R}(c,X^{(n)})]=\mu_{\beta}$, then $P_{G}(c;\beta,X^{(n)})$ is the maximally non-comittal distribution with respect to information different to that contained in this restriction. Tikochinsky established another characteristic that makes the Boltzmann weights and Gibbs distribution an appealing choice as the ASC weighting function: it is maximally stable. If we change our desired resolution level from $\beta_{1}$ to $\beta_{2}$, the change in the induced Gibbs distributions is minimal, in the $L_{2}$ norm sense, among any two distributions $p_{1}$ and $p_{2}$ that satisfy $\mathbb{E}_{p_{1}(c)}[\hat{R}(c,X^{(n)})]=\mu_{\beta_{1}}$ and $\mathbb{E}_{p_{2}(c)}[\hat{R}(c,X^{(n)})]=\mu_{\beta_{2}}$.

We have discussed the role of the resolution parameter $\beta$ in the context of the resolution-stability trade-off, so how can we determine the best value of $\beta$? For this purpose Buhm13 developed the concept of Generalization Capacity which we will first define and then describe in analogy to the Channel Capacity concept of information theory.

Definition 1.5.0.4** (Joint partition function).**

Before we define Generalization Capacity we define the joint partition function between two data sets which measures the equivalent number of hypotheses selected by a weighting function $w_{\beta}(c,X)$ for two different data sets $X^{\prime}$ and $X^{\prime\prime}$:

[TABLE]

Remarks:

1. i.)

We have dropped the superindex (n) for better readability: $X^{\prime}:=X^{\prime(n)}$, $X^{\prime\prime}:=X^{\prime\prime(n)}$, $O^{\prime}:=O^{\prime(n)}$ and $O^{\prime\prime}:=O^{\prime\prime(n)}$. 2. ii.)

As this definition already suggests GC will involve comparing the approximation sets obtained with different data sets of the same size. 3. iii.)

For some pattern analysis problems such as 1.3.0.3 the hypothesis class depends on the object set $O^{(n)}$ so that $\mathcal{C}(O^{\prime})\neq\mathcal{C}(O^{\prime\prime})$. In this case we need a mapping $\psi:\mathcal{O}^{\prime}\rightarrow\mathcal{O}^{\prime\prime}$ so that $\Delta Z_{\beta}(X^{\prime},X^{\prime\prime})$ can be properly defined as $\Delta Z_{\beta}(X^{\prime},X^{\prime\prime})=\sum_{c\in\mathcal{C}(O^{\prime\prime})}w_{\beta}(c,\psi(X^{\prime}))w_{\beta}(c,X^{\prime\prime})$. Although these types of problems are very important in the context of ASC given the infinite VC dimension of the hypothesis class, for the sparse mean estimation problem described in Section 1.2 this is not the case, so we will simply assume, from now on, that $\mathcal{C}:=\mathcal{C}(O^{\prime})=\mathcal{C}(O^{\prime\prime})$. This also means we can dispense with the mappings $\psi$ in this exposition.

Definition 1.5.0.5** (Information Content).**

The information content retrievable from data $(X^{\prime},X^{\prime\prime})$ by a cost function $\hat{R}(c,X)$ with resolution $\beta$ is:

[TABLE]

**Remarks:

**

1. i.

$I_{\beta}$* is a normalized and rescaled version of $\Delta Z_{\beta}(X^{\prime},X^{\prime\prime})$ which measures the equivalent number of selected hypotheses with cost function $\hat{R}(c,X)$ for both data sets $X^{\prime}$ and $X^{\prime\prime}$.* 2. ii.

Since $\lim_{\beta\to 0}Z_{\beta}(X^{\prime})=\lim_{\beta\to 0}Z_{\beta}(X^{\prime\prime})=\lim_{\beta\to 0}\Delta Z_{\beta}(X^{\prime},X^{\prime\prime})=|\mathcal{C}|$ it holds that $\lim_{\beta\to 0}I_{\beta}=0$, which means that for resolution $\beta=0$, where all hypothesis are given a weight of 1, the information content is zero. 3. iii.

Let $\mathcal{C}=\{c_{1},...,c_{|\mathcal{C}}|\}$, $a_{i}=w_{\beta}(c_{i},X^{\prime})\geq 0$ and $b_{i}=w_{\beta}(c_{i},X^{\prime\prime})\geq 0$, then:

[TABLE]

and we can see that $\lim_{\beta\to\infty}I_{\beta}\leq\log|\mathcal{C}|$. This means that for maximum resolution the information content can reach up to the log-size of the hypothesis class.

Definition 1.5.0.6** (Generalization Capacity).**

The generalization capacity of a cost function $\hat{R}(c,X)$ defined over a hypothesis class $\mathcal{C}$ and data space $\mathcal{X}^{n}$ is:

[TABLE]

**Remarks:

**

1. i.

Since $\lim_{\beta\to 0}I_{\beta}=0$ we have that $\lim_{\beta\to 0}I=0$ and,s 2. ii.

since $\lim_{\beta\to\infty}I_{\beta}\leq\log|\mathcal{C}|$ we have that $\lim_{\beta\to\infty}I\leq\log|\mathcal{C}|$

1.6 GC and Shannon’s noisy-channel coding theorem

To motivate the relevance of the Generalization Capacity as an important quantity in itself aswell as a criterion for deciding between cost functions in a pattern analysis problem, we briefly study Shannon’s Noisy-Channel Coding Theorem, the communication protocol suggested therein and the role of Channel Capacity. We then move from the communication context to the pattern learning context and study an analagous learning protocol suggested by Buhm13 where the generalization capacity emerges as a natural counterpart to channel capacity. The exposition of Shannon’s Noisy-Channel Coding Theorem is based on Cover and Yeung.

Shannon’s Noisy-Channel Coding Theorem deals with the rate at which information can be passed through a channel so we first define what information and channels are.

Definition 1.6.0.1** (Shannon Information).**

The Shannon information of an outcome $x$ of a random variable $X\in\mathcal{X}$, where $\mathcal{X}$ is a finite set and $p(x)$ is the probability distribution of $X$ is:

[TABLE]

If we interpret the informativeness of an outcome in terms of the worth of knowing its value the following properties make this a useful measure of information:

1. i.)

$I(x)\geq 0$* $\forall x$* 2. ii.)

Assigns 0 to a certain outcome 3. iii.)

The rarer an outcome the more informative: $p(x)=p,p(y)=q,p<q\Rightarrow I(x)>I(y)$ 4. iv.)

It is continuous in $p(x)$: $\forall\epsilon>0$ $\exists$ $\delta>0:\textrm{ if }|p-q|<\delta\Rightarrow|I(x)-I(y)|<\epsilon$ 5. v.)

Additivity: $p_{XY}(x,y)=p_{X}(x)p_{Y}(y)\Rightarrow I(x,y)=I(x)+I(y)$

Definition 1.6.0.2** (Entropy).**

The entropy of a random variable $X\in\mathcal{X}$, where $\mathcal{X}$ is a finite set, is its expected Shannon Information:

[TABLE]

We can interpret entropy as the average information rate of a random variable. If we want to send messages from a finite message set $\mathcal{W}=\{1,...,M\}$, we may define the information rate of the message set by assuming messages will be sent according to the uniform distribution. In this case:

[TABLE]

We now define joint and conditional entropy.

Definition 1.6.0.3** (Joint Entropy).**

For random variables $X\in\mathcal{X}$ and $Y\in\mathcal{Y}$, with $\mathcal{X}$ and $\mathcal{Y}$ finite sets, the joint entropy of $X$ and $Y$ is defined as:

[TABLE]

Definition 1.6.0.4** (Conditional Entropy).**

For random variables $X\in\mathcal{X}$ and $Y\in\mathcal{Y}$, with $\mathcal{X}$ and $\mathcal{Y}$ finite sets, the joint entropy of $Y$ given $X$ is defined as:

[TABLE]

Conditional entropy is a measure of the mean information left in $Y$ once we know the outcome of $X$. It turns out that $H(X,Y)=H(X)+H(Y|X)$ so the joint entropy can be interpreted as the mean amount of information in $X$ plus the mean amount of information left in $Y$ once the outcome of $X$ is known.

Definition 1.6.0.5** (Mutual Information).**

For random variables $X\in\mathcal{X}$ and $Y\in\mathcal{Y}$, with $\mathcal{X}$ and $\mathcal{Y}$ finite sets, the mutual information between $X$ and $Y$ is defined as:

[TABLE]

Using that $I(X;Y)=H(X)-H(X|Y)$ we can interpret the mutual information as the reduction in information left in $X$ once $Y$ is known (or vice versa). Alternatively, we can interpret $I(X;Y)$ as the information that is common to $X$ and $Y$. If $X$ and $Y$ are independent then they have no information in common and if $X$ depends deterministically on $Y$ then they have the same information.

Definition 1.6.0.6** (Discrete Channel).**

Let $\mathcal{X}$ and $\mathcal{Y}$ be discrete sets and $p(y|x)$ be a transition matrix from $\mathcal{X}$ to $\mathcal{Y}$ that is a valid distribution for all $x$. Then the tuple $(\mathcal{X},p(y|x),\mathcal{Y})$ is a discrete channel where $X\in\mathcal{X}$ and $Y\in\mathcal{Y}$ are the input and output respectively. Remark: We sometimes refer to the discrete channel simply as $p(y|x)$.

If Input $X=x$ is sent through the channel then the output $Y$ is distributed according to $p(y|x)$.

Definition 1.6.0.7** (Discrete Memoryless Channel (DMC)).**

A discrete memoryless channel is a discrete channel $(\mathcal{X},p(y|x),\mathcal{Y})$ such that if a sequence of inputs $X_{1},X_{2},...$ are sent through the channel then:

[TABLE]

Intuitively, the channel forgets all previous communication such that the ouptut of $Y_{t}$ only depends on the input $X_{t}$ and on the distribution $p(y|x)$.

Definition 1.6.0.8** (Capacity of a DMC).**

The capacity of a DMC $(\mathcal{X},p(y|x),\mathcal{Y})$ is defined as:

[TABLE]

Where $X$ and $Y$ are the input and output of the channel.

The capacity of a DMC is a measure of the amount of common information between the input and output in the most optimistic scenario. As we will see later in this section, Shannon’s Noisy-Channel Coding Theorem shows why the capacity of a DMC is an important quantity.

Since a given channel $(\mathcal{X},p(y|x),\mathcal{Y})$ only takes as input $X\in\mathcal{X}$ we need an encoder function to transform our message $w\in\mathcal{W}$ into an acceptable input. If $|\mathcal{X}|=M$ then we may simply assign each message an element of the input set $\mathcal{X}$, however this doesn’t help us avoid errors in communication. If the channel transforms the message such that the output is not the same as the input then an error will occur.

If $|\mathcal{X}|>M$ we have some slack in our input set $\mathcal{X}$ which may help us to avoid errors. Suppose that $\mathcal{X}=\{x_{1},x_{2},x_{3}\}$ and $\mathcal{W}=\{0,1\}$ then we can assign 0 to $x_{1}$ and 1 to $x_{2}$ and $x_{3}$. If an $x_{2}$ is sent through and the channel distorts it into an $x_{3}$ we still avoid error.

If $|\mathcal{X}|\leq M$ we may add slack to our coding scheme by encoding each message $w\in\mathcal{W}$ with a sequence of $n$ symbols $x_{i}\in\mathcal{X}$. In this case we have $|\mathcal{X}|^{n}$ sequences to encode $M$ messages. If we let $n$ grow then we increase the slack in our code and so reduce the probability of error, especially if we assign sets of sequences to each message in a smart way. To prove the Noisy-Channel Coding theorem Shannon constructed such a smart assignment procedure using ideas of typicality which we explore somewhat further on.

We can already touch on how the pattern analysis problem bares some resemblance to the problem of sending a message through a noisy channel: in the former there is some truth or property in nature which is a hypothesis $c\in\mathcal{C}$ and it is encoded in a slack way by a data generating mechanism such that for each hypothesis there correspond many possible data sets $X^{(n)}\in\mathcal{X}^{n}$.

Having broached the idea of slack codes we now define an $(n,M)$ code and give a schematic description of Shannon’s communication scenario.

Definition 1.6.0.9** ($(n,M)$ code).**

An $(n,M)$ code for a DMC $(\mathcal{X},p(y|x),\mathcal{Y})$ is defined by an encoding function $f$ and a decoding function $g$:

[TABLE]

Where:

• •

$\mathcal{W}=\{1,...,M\}$* is called the message set,*

• •

$f(1),...,f(M)\in\mathcal{X}^{n}$* are the codewords and*

• •

$\{f(1),...,f(M)\}$* is the codebook.*

The rate of an $(n,M)$ code is defined as $\frac{1}{n}\log M$. This corresponds to the rate of a uniform random variable over the message set $\mathcal{W}$ divided by $n$ so that it is in the units of bits (or nats depending on the base of the logarithm) per symbol $X\in\mathcal{X}$ and not bits per sequence $X^{(n)}\in\mathcal{X}^{n}$.

A rate $R$ for a DMC is said to be asymptotically achievable if there exists an $(n,M)$ code that for a sufficiently large $n$ can transmit at a rate arbitrarily close to $R$ with arbitrary precision.

Theorem 1.6.0.10** (Shannon’s Noisy-Channel Coding Theorem ).**

A rate $R$ is asymptotically achievable for a DMC $\iff R\leq C$

This theorem justifies the Capacity of a channel as an interesting quantity: it implies that we can achieve error-free communication at a rate equal to the capacity of the DMC. The proof of this part of the theorem involves proposing the $(n,M)$ code shown below and then proving that for an $n$ such that the rate $R=\frac{1}{n}logM$ of the code is close to the capacity of the DMC, the probability of error $P(w\neq\hat{w})$ is small. In this work we are especially interested in the $(n,M)$ code proposed in Shannon’s proof since it forms the basis of a similar coding scheme and communication protocol in which generalization capacity plays an analogous role to that of channel capacity.

Definition 1.6.0.11** (Shannon’s (n,M) code).**

The $(n,M)$ code proposed is the following:

1. 1

Sample $M$ sequences $X^{(n)}$ uniformly at random from $\mathcal{X}^{n}$ and randomly assign each sequence sampled to one of the messages. This establishes the encoding function $f$. Both sender and receiver have the codebook $\{f(1),...,f(M)\}$. 2. 2

Compare the joint entropy of $H(X,Y)$, to the empirical entropy of the the pairs of sequences $(f(1),Y^{(n)}),...,(f(M),Y^{(n)})$. Choose message $i$ such that the empirical entropy of $(f(i),Y^{(n)})$ is close to the entropy $H(X,Y)$. If there is more than one pair of sequences that satisfies this condition decode to $\hat{w}=1$. This establishes the decoding function $g$.

The proof that this $(n,M)$ code can asymptotically achieve a rate $R=C$ involves the concept of typicality which is an application of the Law of Large Numbers. Although we do not give the formal proof we give a sequential illustration of the ideas.

1. i.)

Start with a large $n$ for a slack code. First we choose $n$ large so that we have a lot of slackness in our code, more than we will need, and uniformly at random choose $M$ sequences $X^{(n)}\in\mathcal{X}^{n}$ as our codewords.

2. ii.)

Channel sends messages to non-overlapping regions. We have chosen $n$ so large that even with a lot of noise, when $f(1),...,f(M)\in\mathcal{X}^{n}$ are transformed into $Y_{1}^{(n)},...,Y_{M}^{(n)}\in\mathcal{Y}^{n}$ by the channel $(\mathcal{X},p(y|x),\mathcal{Y})$ the probability that $Y_{i}$ and $Y_{j}$ are close for any $i\neq j$ is essentially zero. The i-th region $\mathcal{R}_{i}$ represents the sequences $Y^{(n)}\in\mathcal{Y}^{n}$ which are jointly typical with $f(i)$. Given that the sequence passed over the channel is $f(i)$, the probability that the sequence received by the decoder is outside this region is essentially zero. If $Y_{j}^{(n)}\in\mathcal{R}_{i}$ and $\nexists k\neq i:Y_{j}^{(n)}\in\mathcal{R}_{k}$ then $g(Y_{j}^{(n)})=i$.

3. iii.)

Decrease $n$ until $\mathcal{Y}^{n}$ is tight around regions. Since $n$ is large the rate $R=\frac{1}{n}logM$ of the code is low. Shannon’s Noisy-Channel Coding Theorem says that we can decrease $n$ so that the rate increases to close to $C$ and the error stays very small. By decreasing $n$ we decrease the size of $\mathcal{Y}^{n}$ so that all the regions $\mathcal{R}_{i}$ are tightly crowded within. If we are at capacity, the overlap between the regions is still essentially zero, but if we make $\mathcal{Y}^{n}$ any smaller by decreasing $n$ further, the overlap will start to grow, meaning the probability of error grows.

As we have already hinted at, the pattern analysis problem can be seen as a special case of the communication problem. We explore this further by proposing the following *Idealized Learning Protocol *:

The communication protocol has the following characteristics:

1. 1

The sender picks $M$ uniformly at random to construct $\mathcal{C}_{M}=\{c_{1},...,c_{M}\}\subseteq\mathcal{C}$: $c_{i}\sim p(c)=\frac{1}{|\mathcal{C}|}$. 2. 2

The sender selects a message $\widetilde{c}_{s}\in\mathcal{C}_{M}$ uniformly at random: $\widetilde{c}_{s}\sim p_{s}(c)=\frac{1}{M}$. Remark: we use the tilde to separate $c_{i}\sim p(c)=\frac{1}{|\mathcal{C}|}$, $i\in\{1,...,M\}$ from $\widetilde{c}_{s}\sim p_{s}(c)=\frac{1}{M}$. 3. 3

The sender and receiver have access to the data generating mechanism $p(x|c)$ which they use to set up the following $(n,M)$ code:

1. a

The encoder function $f$ is constructed by randomly sampling $M$ times a data set of size $n$ from the data generating mechanism $p(x|c)$ to obtain the codebook $\{f(c_{1}),...,f(c_{M})\}$ where $f(c_{i})\in\mathcal{X}^{n}$. Notice that in many problems such as clustering and regression $\mathcal{X}=\mathbb{R}^{d}$ so that $\mathcal{X}^{n}=\mathbb{R}^{n\times d}$, i.e. we code each message using a data matrix. 2. b

Our decoding function $g$ works as in Shannon’s $(n,M)$ code except that $\mathcal{X}$ is not necessarily a finite set so we might need to use joint differential entropy instead of joint entropy. 4. 4

We set up a channel $(\mathcal{X}^{n},p(x^{\prime\prime}|x^{\prime}),\mathcal{X}^{n})$ with our knowledge of the hypothesis and data generating mechanisms $p_{s}(c)$ and $p(x|c)$ respectively:

[TABLE]

Notice that we are assuming that successive sample sets of size $n$ from the data generating mechanism are independent given $c$.

With the exception that we are using what may be a set $\mathcal{X}$ with infinite cardinality to code the message set $\mathcal{C}_{M}=\{c_{1},...,c_{M}\}$ the above Idealized Learning Protocol corresponds to the previous communication protocol. The channel $p(x^{\prime\prime}|x^{\prime})$ characterizes the noisiness of the pattern analysis problem since in a noise-free scenario we would obtain the same data set for each realization of the data generating mechanism, i.e. $X^{\prime}=X^{\prime\prime}$. Since messages are always selected uniformly at random according to $p_{s}(x^{\prime})=p_{s}(c)=\frac{1}{M}$ we may consider the capacity of the channel to be $I(X^{\prime};X^{\prime\prime})$. The capacity of the channel is a measure of the noisiness (the higher the capacity the less is the noise) and is an upper bound on the rate at which any learning algorithm can extract information from specific realizations $X\in\mathcal{X}^{n}$, that generalizes accross realizations, i.e. information about $c$ and not about the noise. In the above Idealized Learning Protocol we can achieve the capacity rate, as before, by choosing a suitable $n$. Since this scenario is highly idealized we make successive changes to it until we arrive at the more useful Learning Protocol proposed by Buhm13 and from which generalized capacity is derived:

1. I

Change expressiveness of codebook instead of size of code sequences. Suppose we can no longer change $n$, the size of our sequence $X\in\mathcal{X}^{n}$, i.e. it is fixed. Instead we are allowed to change $M$ the number of selected to form $\mathcal{C}_{M}$. We can now achieve a rate close to capacity by increasing $M$ instead of decreasing $n$. Our message set $\mathcal{C}_{M}$ and codebook $\{f(c_{1}),...,f(c_{M})\}$ become more expressive as we increase $M$. 2. II

Coding based on transformation set. Additionally, suppose we can only use the data generating mechanism $p(x|c)$ and channel $p(x^{\prime\prime}|x^{\prime})$ once. We still know the form of $p(x|c)$ and $p(x^{\prime\prime}|x^{\prime})$ (and so can calculate entropies for decoding) but can only generate with it once. Furthermore, suppose that we don’t know what hypothesis $c_{D}$ is selected and passed to the data generating mechanism $p(x|c)$. Since we don’t have any information about $c_{D}$ other than the data $X^{\prime}$ and $X^{\prime\prime}$, we use a uniform prior $c_{D}\sim p(c)=\frac{1}{|\mathcal{C}|}$ to describe our uncertainty regarding the true $c_{D}$. All this means we can only generate two data sets $X^{\prime},X^{\prime\prime}\in\mathcal{X}^{n}$: we generate $X^{\prime}$ using $p(x|c)$ and then send it through the channel $p(x^{\prime}|x^{\prime\prime})$ to get $X^{\prime\prime}$. Without access to the data generating mechanism and the true $c_{D}$ we need an alternative way to construct a codebook (i.e. another $f$ for our $(n,M)$ code). Consider the set of unique maps:

[TABLE]

Where $\mathcal{T}=\{t:\mathcal{C}\rightarrow\mathcal{C}\}$.

Notice the following properties about $\mathcal{T}^{h}$:

1. (a)

$|\mathcal{T}^{h}|=|\mathcal{C}|$ 2. (b)

If you apply a fixed $t^{h}\in\mathcal{T}^{h}$ on all $c\in\mathcal{C}$ you get $C$ again. 3. (c)

If you apply all $t^{h}\in\mathcal{T}^{h}$ on a fixed $c\in\mathcal{C}$ you get $C$ again.

Additionally, consider the set of maps:

[TABLE]

We assume that we have a mapping $\phi:\mathcal{T}^{h}\rightarrow\mathcal{T}^{D}$ such that for a given $t^{h}\in\mathcal{T}^{h}$, $\phi(t^{h})=t^{D}\in\mathcal{T}^{D}$ and:

[TABLE]

Where the posteror $p(c|x)$ is obtained from the data generating mechanism $p(x|c)$ and the prior $p(c)$:

[TABLE]

This means that if data set $X$ is generated under hypothesis $c$ with the data generating mechanism $p(x|c)$ then, for a given move $t^{h}$ within the hypothesis space $\mathcal{C}$ we know how to make a corresponding move $t^{D}$ in the coding/data space $\mathcal{X}^{n}$. The assumption that we can obtain a mapping $\phi$ is reasonable in some contexts such as in the sparse mean estimation problem that is the main topic of this work. In other pattern analysis problems such as in the mean estimation problem 1.3.0.1 where the hypothesis class $\mathcal{C}$ has infinite cardinality, the validity of this assumption is not clear.

With the above assumption we will be able to encode $M$ $c_{i}\in\mathcal{C}_{M}$ into a codebook $\{f(c_{1}),...,f(c_{M})\}\subseteq\mathcal{X}^{n}$ However we can only pass one data set through the channel $p(x^{\prime\prime}|x^{\prime})$ and arbitrarily choose to pass $f(c_{D})=X^{\prime}=x^{\prime}_{0}$ which gives the random output $X^{\prime\prime}=x^{\prime\prime}_{0}$. Observe that:

[TABLE]

Where we have used assumption 1.6.0.12 and the properties of 1.6.0.10. This implies that:

[TABLE]

We wish to mimic the channel $p(x^{\prime\prime}|x^{\prime})$ by mimicking the noise process that contaminated $x^{\prime}_{0}$ to produce $x^{\prime\prime}_{0}$. We may think of the output of the channel $p(x^{\prime\prime}|x^{\prime})$ as a function $\alpha$ of the input $x^{\prime}$ and a noise realization of some $N$. We then have that

[TABLE]

Since we have passed $x^{\prime}_{0}$ through the channel and have observed $X^{\prime\prime}=x^{\prime\prime}_{0}$ we implicitly have a noise observation $N=n_{0}$. Although, we can’t pass $t^{D}_{i}(x^{\prime}_{0})$ through the channel to get $t^{D}_{i}(X^{\prime\prime})$ (recall we are only allowed to use the channel once, and we have already used it to pass $x_{0}^{\prime}$ through), we can mimic the output with $t_{i}^{D}(x^{\prime\prime}_{0})$ which is the result of evaluating $\alpha$ on $n_{0}$ instead of on a new realization of $N$:

[TABLE]

With the set $\mathcal{T}^{c}$ and the mapping $\phi$ we have the necessary elements to replace our encoding function $f$. Incorporating the changes to the encoding function and channel, the learning protocol, thus far, consists of:

1. 1

The sender picks $M$ transformations uniformly at random to construct $\mathcal{T}_{M}^{h}=\{t^{h}_{1},...,t^{h}_{M}\}\subseteq\mathcal{T}^{h}$: $t^{h}_{i}\sim p(t^{h})=\frac{1}{|\mathcal{T}^{h}|}=\frac{1}{|\mathcal{C}|}$. Even though we don’t know what $c_{D}$ is we can set up our message set $C_{M}$ using $\mathcal{T}_{M}^{h}$: $C_{M}=\{c_{1},...,c_{M}\}=\{t^{h}_{1}(c_{D}),...,t^{h}_{M}(c_{D})\}$. In fact, we may now say that $\mathcal{T}^{h}_{M}$ is the message set. 2. 2

The sender selects a message $\widetilde{t}^{h}_{s}\in\mathcal{T}^{h}_{M}$ uniformly at random: $\widetilde{t}_{s}^{h}\sim p_{s}(t^{h})=\frac{1}{M}$. Remark: we use the tilde to separate $t_{i}^{h}\sim p(t^{h})=\frac{1}{|\mathcal{C}|}$, $i\in\{1,...,M\}$ from $\widetilde{t}_{s}^{h}\sim p_{s}(t^{h})=\frac{1}{M}$. 3. 3

The sender and receiver have access to the transformation set $\mathcal{T}^{h}_{M}$ and the mapping $\phi$ which they use to set up the following $(n,M)$ code:

1. a

The encoder function $f$ is constructed by applying $\phi$ to each $t^{h}_{i}\in\mathcal{T}_{M}^{h}$ to construct $\mathcal{T}_{M}^{D}=\{t^{D}_{1},...,t^{D}_{M}\}=\{\phi(t^{h}_{1}),...,\phi(t^{h}_{M})\}$. Our codebook vector is then $\{f(c_{1}),...,f(c_{M})\}=\{f(t^{h}_{1}(c_{D})),...,f(t^{h}_{M}(c_{D}))\}=\{t_{1}^{D}(X^{\prime}),...,t_{M}^{D}(X^{\prime})\}$. 2. b

Since we still know the distribution $p(x|c)$ we may use it to calculate $p(x^{\prime},x^{\prime\prime})=\frac{1}{M}\sum_{c\in\mathcal{C}}p(x^{\prime\prime}|c)p(x^{\prime\prime}|c)$. Notice that since we know how $\widetilde{c}_{s}=\widetilde{t}_{s}^{h}(c_{D})$ is selected we use $p_{s}(c)=\frac{1}{M}$. With $p(x^{\prime},x^{\prime\prime})$ we can calculate the joint entropy $H(X^{\prime},X^{\prime\prime})$ and use it for decoding as before. 4. 4

We can only send one data set $X^{\prime}=x^{\prime}_{0}$ through the channel $(\mathcal{X}^{n},p(x^{\prime\prime}|x^{\prime}),\mathcal{X}^{n})$ and so can only observe one output $X^{\prime\prime}=x^{\prime\prime}_{0}$. However, we may mimic the behavior of the channel for other input data sets $t_{i}^{D}(X^{\prime})$ by using, as derived above, that:

[TABLE]

This means that we may mimic the channel $p(x^{\prime\prime}|x^{\prime})$ by outputing $t^{D}_{i}(x^{\prime\prime}_{0})$ for a given input $t^{D}_{i}(x^{\prime}_{0})$. This output corresponds to the output the actual channel $(\mathcal{X}^{n},p(x^{\prime\prime}|x^{\prime}),\mathcal{X}^{n})$ would have given for an input $t^{D}_{i}(x^{\prime}_{0})$, assuming the same realization of the noise process as ocurred when $X^{\prime}$ passed through the channel. Since the noise realization for one data set, is made up of $n$ components, the hope is that the realization observed summarizes the noisiness of the channel. In other words we hope that applying this noise realization to any input data set $t^{D}_{i}(x^{\prime}_{0})$, the output data set $t^{D}_{i}(x^{\prime\prime}_{0})$ is similar to that we would get by passing $t^{D}_{i}(X^{\prime})$ through the real channel (to which we no longer have access).

The assumption that we still know the distribution $p(x|c)$ is the last idealized, unrealistic element of our learning protocol. The last change to the protocol involves dispensing with this assumption. To do so we will use the Boltzmann approximation sets discussed in sections 1.4 and 1.5. 3. III

Decoding based on approximation sets. Finally suppose we don’t know the distribution of the data generating mechanism $p(x|c)$ or that of the channel $p(x^{\prime\prime}|x^{\prime})$. This means we need a new decoding function $g$ since we don’t know $p(x^{\prime},x^{\prime\prime})$ and so cannot use the joint entropy $H(X^{\prime},X^{\prime\prime})$ for decoding. This is where our learning algorithm comes to the fore in the form of the cost function $\hat{R}(c,X)$ and the Gibbs distributions corresponding to both data sets: $P_{G}(c;\beta,X^{\prime})$ and $P_{G}(c;\beta,X^{\prime\prime})$. We first describe the new decoding function $g$ and then discuss the ideas behind it and its relationship to genealization capacity.

Let

[TABLE]

Then the decoding rule is

[TABLE]

Where ties are resolved by taking the minimum $j$. Before discussing how to choose the resolution parameter $\beta$ we can show the final Learning Protocol schematic.

What $\beta$ should we use? In general we can use any $\beta$, however to find the channel capacity we must choose it so that for a given $M$ (which determines the rate $R=\frac{1}{n}\log M$, given $n$ is fixed) we can achieve error-free communication. We then increase $M$ to $M^{*}$ such that if we increase it any further there exists no $\beta$ that allows error-free communication. This $M^{*}$ determines the maximum achievable rate of our code. We illustrate the process of finding the maximum achievable rate of the above $(n,M)$ code.

1. i

Start with low expressiveness $M$ for slack code. We uniformly sample $M$ $c_{i}\in\mathcal{C}_{M}$ by sampling $M$ transformations $t_{i}^{h}\in\mathcal{T}^{h}_{M}$ and use the corresponding set $\mathcal{T}^{D}_{M}$ to build our codebook. We select a transformation $\widetilde{t}^{h}_{s}$ and pass the econcoded message $\widetilde{t}^{D}_{s}(X^{\prime})$ through the channel to decoder that receives $\widetilde{t}^{D}_{s}(X^{\prime\prime})$.

2. ii

Create high resolution $\beta$ approximation sets. Using high resolution $\beta$, we calculate $M$ approximation sets $w_{\beta}(c,t_{j}^{D}(X^{\prime}))$, one for each codeword in the codebook. Using the received data set, we calculate an additional approximation set $w_{\beta}(c,\widetilde{t}_{s}^{D}(X^{\prime\prime}))$, and apply decoding rule. In the case illustrated in figure 1.10 $\Delta Z_{\beta}^{j}=0$ $\forall j$ so by default we decode message to $t_{1}^{h}(c_{D})$. The red circles represent the approximation sets $w_{\beta}(c,t_{j}^{D}(X^{\prime}))$ while the blue circle represents the approximation set $w_{\beta}(c,\widetilde{t}_{s}^{D}(X^{\prime\prime}))$. Recall from Section 1.4 that although, strictly speaking, $w_{\beta}(c,X)$ is a weight vector over the entire hypothesis class $\mathcal{C}$, the circles represent the subset of $\mathcal{C}$ where the majority of the weight is supported.

3. iii

Lower resolution $\beta$ to increase intersection. Since the current resolution level doesn’t allow significant intersection between $w_{\beta}(c,\widetilde{t}_{s}^{D}(X^{\prime\prime}))$ and any of the $w_{\beta}(c,t_{j}^{D}(X^{\prime}))$ approximation sets we lower the resolution level until there is some intersection meaning we can decode the message without error. Since the hypothesis class $\mathcal{C}$ is not cluttered with approximation sets, because we are not near capacity, the intersection need not represent a large percentage of the respective approximation sets.

4. iv

Increase expressiveness $M$, while adjusting resolution $\beta$. In finding the maximum rate of our code we increase the size of the message set which means more approximation sets over $\mathcal{C}$. However, as happens in part (b) of figure 1.12, for a given resolution $\beta$ the approximation set $w_{\beta}(c,\widetilde{t}_{s}^{D}(X^{\prime\prime}))$ intersects with more than one approximation set $w_{\beta}(c,t_{j}^{D}(X^{\prime}))$ meaning the probability of decoding the wrong message increases. We can, as is shown on part (c), fix this by decreasing the resolution but this increases the size of the approximation sets $w_{\beta}(c,t_{j}^{D}(X^{\prime}))$ so that they intersect between themselves. This also increases the probability of error since the set $w_{\beta}(c,\widetilde{t}_{s}^{D}(X^{\prime\prime}))$ intersects with several $w_{\beta}(c,t_{j}^{D}(X^{\prime}))$ sets simultaneously. Finally in part (d) we obtain obtain the maximum rate of our code: if we add any more approximation sets $w_{\beta}(c,t_{j}^{D}(X^{\prime}))$, they will become cluttered between themselves (we go back to the situation in part c) and if we then fix this by increasing resolution, more than one of these sets will intersect significantly with $w_{\beta}(c,\widetilde{t}_{s}^{D}(X^{\prime\prime}))$ (we go back to the situation in part b).

We have arrived at a fundamental trade-off between, on the one hand, the expressiveness of our code (the cardinality of $\mathcal{T}^{h}_{M}$) and the resolution of our decoding mechanism (the parameter $\beta$) and on the other, the stability of the learning protocol measured by the probability of error, itself a function of the quality of the overlap $\max_{j\in\{1,...,M\}}\Delta Z_{\beta}^{j}$.

Suppose the sender sends the hypothesis $c_{j}=t_{j}^{h}(c_{D})$ corresponding to the dataset $t_{j}^{D}(X^{\prime})$. We want to compare the approximation sets $w_{\beta}(c,t_{j}^{D}(X^{\prime}))$ and $w_{\beta}(c,t_{j}^{D}(X^{\prime\prime}))$ to get an idea of what needs to happen to be able to achieve the maximum learning rate. For a given resolution $\beta$:

1. 1

We want to be able to increase the size of the message set, taking care that the approximation sets $w_{\beta}(c,t_{j}^{D}(X^{\prime}))$ don’t become cluttered. We can achieve this by maximizing:

[TABLE]

This criterion will tend to make $Z_{\beta}(t_{j}^{D}(X^{\prime}))$ small. 2. 2

We want to make sure that the quality of the overlap $\Delta Z_{\beta}(t^{D}_{j}(X^{\prime}),t^{D}_{j}(X^{\prime\prime}))$ is good. However we need a relative measure since low resolution communication will in general lead to a bigger $\Delta Z_{\beta}(t^{D}_{j}(X^{\prime}),t^{D}_{j}(X^{\prime\prime}))$. We can achieve this by maximizing:

[TABLE]

This criterion will tend to make $Z_{\beta}(t_{j}^{D}(X^{\prime\prime}))$ small and $\Delta Z_{\beta}(t^{D}_{j}(X^{\prime}),t^{D}_{j}(X^{\prime\prime}))$ large.

Now notice that for a given $X^{\prime}$, $X^{\prime\prime}$ and $\beta$ these two quantities don’t depend on the transformation $t^{D}_{j}$ due to assumption 1.6.0.12 so that we may assume without loss of generality that $t_{j}^{h}(c)=c$ and $t_{j}^{D}(X)=X$. So for a given $X^{\prime}$ and $X^{\prime\prime}$ the bigger $\frac{|C|}{Z_{\beta}(X^{\prime})}$ and $\frac{\Delta Z_{\beta}(X^{\prime},X^{\prime\prime})}{Z_{\beta}(X^{\prime\prime})}$ the larger the learning rate will be. This gives us a qualitative notion of why generalization capacity defines the maximum learning rate of our learning algorithm.

We now analyze the probability of error of the above Learning Protocol to more formally understand the role of generalization capacity.

Recall what the random quantities in the Learning Protocol are:

• •

$X^{\prime}$,$X^{\prime\prime}\sim p(x|c)$

• •

$\mathcal{T}^{h}_{M}=\{t^{h}_{1},...,t^{h}_{M}\}$, where $t^{h}_{j}\sim p(t^{h})=\frac{1}{|\mathcal{C}|}$

• •

$\widetilde{t}^{h}_{s}\sim p_{s}(t^{h})=\frac{1}{M}$

Now let $J_{s^{-}}:=\{1,...,M\}\setminus\{s\}$. We have that:

[TABLE]

Where we have used the union bound to establish the inequality. Note that $\frac{\Delta Z_{\beta}^{j}}{\Delta Z_{\beta}^{s}}$ is a random variable depending on $X^{\prime}$, $X^{\prime\prime}$, $\mathcal{T}^{h}_{M}$ and $\widetilde{t}^{h}_{s}$. Using the Markov inequality and the independence of $(X^{\prime},X^{\prime\prime})$ and $t^{h}_{j}$ we see the following:

[TABLE]

Where:

• •

$(1)$ is due to the Markov inequality,

• •

$(2)$ is due to the fact that $\widetilde{t}_{s}^{h}$ is given so there is no need to integrate over it,

• •

$(3)$ is due to the chain rule of probability,

• •

$(4)$ is due to the fact that $\Delta Z_{\beta}^{j}$ is independent of $t^{h}_{i}$ for $i\neq j$,

• •

$(5)$ and $(7)$ are due to the fact that $(X^{\prime},X^{\prime\prime})$ is independent of $t_{j}^{h}$,

• •

$(6)$ is due to the fact that $\Delta Z_{\beta}^{s}$ is independent of $t_{j}^{h}$ for $j\neq s$,

• •

$(8)$ is due to the linearity of expectations, and

• •

$(9)$ is due to the fact that $t_{j}^{h}$ are identically distributed for $j\in\{1,...,M\}$.

Now

[TABLE]

Where for the last equivalence we have used assumption 1.6.0.12. Using the properties of 1.6.0.10 we can establish the following:

[TABLE]

So we have that

[TABLE]

Again using the properties of 1.6.0.10 we have that:

[TABLE]

with which

[TABLE]

So error free learning is possible as long as, on average, for data sets $X^{\prime}$ and $X^{\prime\prime}$ it holds that $\log(M-1)<I_{\beta}$.

Finally we mention that since the generalization capacity defines the maximum learning rate of a cost function $\hat{R}(c,X)$ it can be used as a criterion for deciding which cost function to use: simply use the cost function with highest generalization capacity. We will see an application of this in Section 3.6.

Chapter 2 Mean localization

2.1 Generalization capacity

Recall from Section 1.4 that the empirical risk function $\hat{R}(\mu,X)$ is defined with respect to a loss function $\rho(\mu,X)$. We will mostly deal with the square loss function:

[TABLE]

We calculate the empirical risk, which is our cost function in the context of ASC, with respect to this loss function.

[TABLE]

With which we can define the cost function as:

[TABLE]

Recall that in the problem at hand $\mathcal{C}=\mathbb{B}^{d}$ so that:

[TABLE]

Where:

• •

$Z_{\beta}(X)=\sum_{\mu\in\mathbb{B}^{d}}e^{-\beta\hat{R}(\mu,X)}=\sum_{\mu\in\mathbb{B}^{d}}e^{-\beta||\mu-\bar{X}||^{2}_{2}}$ and

• •

$\Delta Z_{\beta}(X^{\prime},X^{\prime\prime})=\sum_{\mu\in\mathbb{B}^{d}}e^{-\beta\{\hat{R}(\mu,X^{\prime})+\hat{R}(\mu,X^{\prime\prime})\}}=\sum_{\mu\in\mathbb{B}^{d}}e^{-\beta\{||\mu-\bar{X}_{1}||^{2}_{2}+||\mu-\bar{X}_{2}||^{2}_{2}\}}$

Since we don’t know the distribution of $Z_{\beta}(X)$ or $\Delta Z_{\beta}(X^{\prime},X^{\prime\prime})$ we will use simulation to estimate $I$. Before describing the simulation algorithm notice that $I$ is a function of the two data set means $\bar{X}_{1}$ and $\bar{X}_{2}$ and since $X$ has a normal distribution:

[TABLE]

If we let $\xi=\frac{\bar{X}-\mu^{0}}{\nicefrac{{\sigma}}{{\sqrt{n}}}}\sim N(0,I_{d})$ then $\bar{X}=\mu^{0}+\frac{\sigma}{\sqrt{n}}\xi$ and we have that:

[TABLE]

Where:

• •

$\hat{R}(\mu,\xi)=||\mu-\mu^{0}-\frac{\sigma}{\sqrt{n}}\xi||^{2}_{2}$,

• •

$w_{\beta}(\mu,\xi)=e^{-\beta\hat{R}(\mu,\xi)}$

• •

$P_{G}(\mu;\beta,\xi)=\frac{w_{\beta}(\mu,\xi)}{\sum_{\mu\in\mathbb{B}^{d}}w_{\beta}(\mu,\xi)}$

• •

$Z_{\beta}(\xi)=\sum_{\mu\in\mathbb{B}^{d}}e^{-\beta||\mu-\mu^{0}-\frac{\sigma}{\sqrt{n}}\xi||^{2}_{2}}$ and

• •

$\Delta Z_{\beta}(\xi_{1},\xi_{2})=\sum_{\mu\in\mathbb{B}^{d}}e^{-\beta\{||\mu-\mu^{0}-\frac{\sigma}{\sqrt{n}}\xi_{1}||^{2}_{2}+||\mu-\mu^{0}-\frac{\sigma}{\sqrt{n}}\xi_{2}||^{2}_{2}\}}$

Using this alternate expression we can reduce the number of simulations by a factor of $n$. Notice that varying both $n$ and $\sigma$ doesn’t make sense since $\frac{\sigma^{2}}{n}$ characterizes the variance of the data set. For this reason we leave $n=100$ fixed and only vary $\sigma$ in our simulation experiments.

2.2 Exhaustive algorithm

To estimate the generalization capacity for the mean localization problem, for a given $\mu^{0}$ and $\sigma$, take the following steps:

Choose a grid of relevant $\beta$ values: $\underline{\beta}=(\beta_{1},...,\beta_{l})$ 2. 2.

For $i=1$ to $m$

1. a.

Simulate $\xi_{1}^{i},\xi_{2}^{i}\sim N(0,I_{d})$ 2. b.

For $k=1$ to $l$

• •

Calculate information content:

[TABLE] 3. 3.

For $k=1$ to $l$

• •

Estimate mean information content:

[TABLE]

• •

Estimate generalization capacity:

[TABLE]

Remark. When dealing with real data we don’t know what $\mu^{0}$ is. This doesn’t matter since GC is independent of any particular hypothesis, rather it depends on the data generating mechanism and on the cost function $\hat{R}(\mu,X)$. Provided we can simulate from the model that we assume generated the data we will always be able to estimate GC by simulating from the model for an arbitrary set of parameters $\mu^{0}$. By calculating GC we obtain a $\beta^{*}$ that resolves the resolution-stability trade off. We may then use $\beta^{*}$ on the real data to obtain an appropriate approximation set of hypotheses. Alternatively we may compare the GC associated to different cost functions and choose the cost function with the highest GC.

2.3 Simulation results

We used the following parameters for the simulation experiments:

• •

$d=8$,

• •

$m=200$,

• •

100 different $\beta$ values from 0.01 to 20, and

• •

30 different noise levels $\sigma$ from 0.1 to 10.

To check simulation results made sense we first calculated the Gibbs distribution $P_{G}(c;\beta^{*},\xi)$ distribution over $\mathcal{C}=\mathbb{B}^{d}$, where $\beta^{*}$ is the resolution parameter that allows generalization capacity to be reached. Since $d=8$ $P_{G}(c;\beta^{*},\xi)$ corresponds to a vector with $2^{8}=258$ entries. To display the results in an easy to read fashion we aggregated this vector to produce a component-wise Gibbs distribution:

[TABLE]

The following graph is an illustration of the $\mathbb{P}_{G}(\mu^{0}_{j}=1|\beta^{*})$ estimate for different noise levels.

The blue dots show the true value of $\mu^{0}$ for each of its components. The lower the noise $\sigma$ the less uncertainty about the value of the $\mu^{0}$ we have. We next show the average information content for different resolutions $\beta$ and noise levels $\sigma$.

The crossed circles represent the pairs $(\beta^{*},I)$ where average information content is maximized. The lower the noise level the higher the generalization capacity is. For low noise levels, sucha as $\sigma=0.1$, we can obtain gains in average information content the higher the resolution $\beta$ (albeit at a diminishing rate) i.e. generalization capacity is basically an increasing function of resolution for these noise levels. This means that for such low noise we can let $\beta\rightarrow\infty$ and obtain the empirical risk minimizer. For medium range noise levels, such as $\sigma=4.9$, once we go past the resolution threshold $\beta^{*}$, the average information content decreases dramatically i.e. generalization capacity is a convex function of resolution for these noise levels. Here we see the resolution-stability trade-off clearly. For this level of noise we can only decrease our approximation sets to a certain size parametrized by $\beta^{*}$ before we start to get very unstable sets with little information.

We now show the generalization capacity for different noise levels.

The blue line shows the generalization capacity while the red line shows the true Gibbs probabilty, i.e. $P_{G}(\mu^{0};\beta^{*},\xi)$. As expected the generalization capacity decreases toward zero as the noise level becomes so big as to completely drown out the signal $\mu^{0}$. Notice that for noise levels $\sigma<1.3$, $P_{G}(\mu^{0};\beta^{*},\xi)=1$ and $I=d\log 2=8\log 2\approx 5.55$ indicating that we can completely recover the signal $\mu^{0}$ with the empirical risk minimizer ( $I=d\log 2\Rightarrow Z_{\beta}(\xi_{1})Z_{\beta}(\xi_{2})=\Delta Z_{\beta}(\xi_{1},\xi_{2})$!).

2.4 Log-sum-exp trick

We can express the information content as

[TABLE]

In turn, we can express the log partition functions as:

[TABLE]

where:

• •

$a_{i}=-\beta\hat{R}(\xi,\mu_{i})<0$,

• •

$b_{i}=-\beta(\hat{R}(\xi_{1},\mu_{i})+\hat{R}(\xi_{2},\mu_{i}))<0$ and

• •

$\mathcal{C}=\{\mu_{1},...,\mu_{|\mathcal{C}|}\}$

When implementing the algorithm of Section 2.2 we may run into the problem that for very high resoluton values $\beta$, $\sum_{i=1}^{\mathcal{C}}e^{a_{i}}$ and $\sum_{i=1}^{\mathcal{C}}e^{b_{i}}$ are so small that they are represented as 0 using limited-precision, floating point numbers. This means $\log Z_{\beta}(\xi)$ and $\log\Delta Z_{\beta}(\xi_{1},\xi_{2})$ are represented as $-\infty$ with which our calculation of $I_{\beta}$ breaks down. To solve this underflow problem we use the log-sum-exp trick. Let:

[TABLE]

Then,

[TABLE]

If we calculate $\log Z_{\beta}(\xi)$ using the last expression, this solves the underflow problem since $\sum_{i=1}^{\mathcal{C}}e^{a_{i}-A}>1$ so that $\log\sum_{i=1}^{\mathcal{C}}e^{a_{i}-A}>0$.

2.5 Variance reduction with common random numbers

If we want to estimate by simulation:

[TABLE]

We can either:

1. a.

Generate $2m$ i.i.d. realizations of $X$: $X_{1}=\{x_{11},...,x_{1m}\}$,$X_{2}=\{x_{21},...,x_{2m}\}$. Then:

[TABLE]

or we can, 2. b.

Generate $m$ i.i.d. realizations of $X$: $X_{1}=\{x_{11},...,x_{1m}\}$. Then:

[TABLE]

The second method is an example of Common Random Numbers (CRNs) since we use the same pseudo-random numbers to estimate $\mathbb{E}_{X}[f(X)]$ and $\mathbb{E}_{X}[g(X)]$.

CRNs is a variance reduction technique, although, strictly speaking, it does not always succeed in reducing the variance of an estimator. To see when it might succeed we compare the variance of $\hat{\mu}_{f}^{1}-\hat{\mu}_{g}^{2}$ and $\hat{\mu}_{f}-\hat{\mu}_{g}$:

[TABLE]

[TABLE]

If $f$ and $g$ are both either monotonically non-decreasing or mononotonically non-increasing then:

[TABLE]

We now obtain a different expression for $Z_{\beta}(\xi)$ and $\Delta Z_{\beta}(\xi_{1},\xi_{2})$ in order to use the CRN technique for the estimation of $\mathbb{E}_{(\xi_{1},\xi_{2})}[I_{\beta}]$. We start with an alternative, but equivalent (proportional in $\mu$) cost function $\hat{R}(\mu,X)$. Using that $\mu_{j}\in\{0,1\}$ we have that:

[TABLE]

So that \mu^{T}(1-2\bar{x})\mathrel{\overset{\makebox[0.0pt]{\mbox{\tiny\mu}}}{\propto}}||\mu-\bar{x}||^{2}_{2}.

Again, using that $\bar{X}=\frac{\sigma}{\sqrt{n}}\xi+\mu^{0}$ we have that:

[TABLE]

Since $\frac{1}{\sqrt{2}}(\xi_{1}+\xi_{2})\sim N(0,I_{d})$ we have that:

[TABLE]

Where:

• •

$\hat{R}(\mu,\xi):=\mu^{T}(1-2\frac{\sigma}{\sqrt{n}}\xi-2\mu^{0})$

• •

$\hat{R}^{\prime}(\mu,\xi):=2\mu^{T}(1-\frac{\sigma}{\sqrt{n}}\sqrt{2}\xi-2\mu^{0})$

Using these expressions we implemented 3 different CRN algorithms to estimate $I_{\beta}$ and $I$, and compared the variance estimate $\hat{\mathbb{V}}_{(\xi_{1},\xi_{2})}[\hat{I}_{\beta^{*}}]$ of each, where $\beta^{*}=\underset{\beta\in\{\beta_{1},...,\beta_{l}\}}{\operatorname{arg}\,\operatorname{max}}\;\hat{\mathbb{E}}_{(\xi_{1},\xi_{2})}[\hat{I}_{\beta}]$.

All three algorithms generate $2m$ realizations of $\xi\sim N(0,I_{d})$ but differ in wether they use them for the estimation of $\mathbb{E}_{\xi}[\log Z_{\beta}(\xi)]$, $\mathbb{E}_{(\xi_{1},\xi_{2})}[\log\Delta Z_{\beta}(\xi_{1},\xi_{2})]=\mathbb{E}_{\xi}[\log\Delta Z_{\beta}(\xi)]$ or both:

CRN-1 Generate $2m$ realizations of $\xi\sim N(0,I_{d})$. Use first $m$ to calculate $\hat{\mathbb{E}}_{\xi_{1}}[\log Z_{\beta}(\xi_{1})]$ and $\hat{\mathbb{E}}_{\xi_{2}}[\log Z_{\beta}(\xi_{2})]$ and second $m$ to calculate $\hat{\mathbb{E}}_{\xi}[\log\Delta Z_{\beta}(\xi)]$. 2. 2.

CRN-2 Generate $2m$ realizations of $\xi\sim N(0,I_{d})$. Use all $2m$ to calculate $\hat{\mathbb{E}}_{\xi_{1}}[\log Z_{\beta}(\xi_{1})]$, $\hat{\mathbb{E}}_{\xi_{2}}[\log Z_{\beta}(\xi_{2})]$ and $\hat{\mathbb{E}}_{(\xi_{1},\xi_{2})}[\log\Delta Z_{\beta}(\xi_{1},\xi_{2})]$. 3. 3.

CRN-3 Generate $2m$ realizations of $\xi\sim N(0,I_{d})$. Use first $m$ to calculate $\hat{\mathbb{E}}_{\xi_{1}}[\log Z_{\beta}(\xi_{1})]$, second $m$ to calculate $\hat{\mathbb{E}}_{\xi_{2}}[\log Z_{\beta}(\xi_{2})]$ and all $2m$ to calculate $\hat{\mathbb{E}}_{(\xi_{1},\xi_{2})}[\log\Delta Z_{\beta}(\xi_{1},\xi_{2})]$. This CRN algorithm is actually the algorithm proposed in Section 2.2. We simulate $(\xi_{1},\xi_{2})$ $m$ times and use it to estimate both $\mathbb{E}_{(\xi_{1},\xi_{2})}[\log Z_{\beta}(\xi_{1})Z_{\beta}(\xi_{2})]$ and $\mathbb{E}_{(\xi_{1},\xi_{2})}[\log\Delta Z_{\beta}(\xi_{1},\xi_{2})]$.

We used the following parameters for the simulation experiments:

• •

$d=8$,

• •

$m=100$,

• •

100 different $\beta$ values from 0.01 to 20, and

• •

30 different noise levels $\sigma$ from 0.1 to 4.

Figure 2.4(b) shows the estimation of the generalization capacity $\hat{I}=\max_{\beta\in\mathbb{R}^{+}}\hat{\mathbb{E}}_{(\xi_{1},\xi_{2})}\log\hat{I}_{\beta}$ and the square root of variance $\sqrt{\hat{\mathbb{V}}_{(\xi_{1},\xi_{2})}[\hat{I}_{\beta^{*}}]}$ using all 3 methods.

Clearly method CRN-3, the method described in Section 2.2, has the least variance. This makes sense because if $\xi_{1},\xi_{2},\xi_{3}$ are i.i.d.:

1. a.

$Cov(\log Z_{\beta}(\xi_{1}),\log Z_{\beta}(\xi_{1}))>0\Rightarrow\mathbb{V}[\log Z_{\beta}(\xi_{1})+\log Z_{\beta}(\xi_{1})]>\mathbb{V}[\log Z_{\beta}(\xi_{1})+\log Z_{\beta}(\xi_{2})]$, so using the same random number to estimate $Z_{\beta}(\xi_{1})$ and $Z_{\beta}(\xi_{2})$ actually increases the variance, while, 2. b.

$Cov(\log Z_{\beta}(\xi_{1}),\log\Delta Z_{\beta}(\xi_{1},\xi_{2}))>0\Rightarrow\mathbb{V}[\log\Delta Z_{\beta}(\xi_{1},\xi_{2})-\log Z_{\beta}(\xi_{1})]<\mathbb{V}[\log\Delta Z_{\beta}(\xi_{1},\xi_{2})-\log Z_{\beta}(\xi_{3})]$, so using the same random number to estimate $\Delta Z_{\beta}(\xi_{1},\xi_{2})$ and $Z_{\beta}(\xi_{1})$ decreases the variance.

Chapter 3 Sparse mean localization

3.1 Exhaustive algorithm

For the sparse mean localization problem the hypothesis space is restricted to binary vectors with $k$ entries equal to one:

[TABLE]

Notice that $|\mathcal{C}^{k}|={d\choose k}$. In terms of the algorithm in Section 2.2, the only thing that changes is that the sums involved in $Z_{\beta}(\xi)$ and $\Delta Z_{\beta}(\xi_{1},\xi_{2})$ are over $\mathcal{C}^{k}$ instead of the entire $\mathcal{C}=\mathbb{B}^{d}$.

In the non-sparse case, to generate the vectors $\mu_{i}$ such that $\mathcal{C}=\{\mu_{1},...,\mu_{2^{d}}\}$ we can simply use the mapping $\kappa:\{0,...,2^{d}-1\}\rightarrow\mathcal{C}$ where $\kappa$ maps a positive integer $0\leq i\leq 2^{d}-1$ to its $d$ length binary representation.

In the sparse case, to generate the vecors $\mu_{i}$ such that $\mathcal{C}^{k}=\{\mu_{1},...,\mu_{{d\choose k}}\}$ we use the mapping $\gamma:\{1,...,{d\choose k}\}\rightarrow\mathcal{C}^{k}$ where $\gamma$ maps a positive integer $1\leq i\leq{d\choose k}$ to the i-th element of $\mathcal{C}^{k}$, assuming that $\mathcal{C}^{k}$ is in lexicographical order.

The algorithm shown below, adapted from Lehmer (pp. 27-29), can be used to produce the mapping $\gamma$. It is based on the fact that if $\mathcal{C}^{k}$ is in reverse lexicographical order then we can represent the number $1\leq i\leq{d\choose k}$ as:

[TABLE]

Where $p_{j}\in\{1,...,d\}$ gives the position of the j-th entry of $\mu_{i}$ that is equal to one. i.e. $\mu_{ip_{j}}=1$ $\forall j\in\{1,...,k\}$.

Algorithm to obtain $\mu_{i}$

Initalize:

• •

Set $\mu_{ij}\leftarrow 0$ $\forall j\in\{1,...,d\}$.

• •

Set $m\leftarrow i-1$. 2. 2.

For $j$ from $k$ to $1$:

1. a.

$p_{j}\leftarrow\underset{l\in\{t:{t\choose j}\leq m\}}{\operatorname{arg}\,\operatorname{max}}\;{l\choose j}$ 2. b.

$m\leftarrow m-{p_{j}\choose j}$ 3. c.

$\mu_{ip_{j}}\leftarrow 1$

3.2 Simulation results: exhaustive algorithm

We used the following parameters for the simulation experiments:

• •

$d=10$,

• •

$k=1,2,3,4,5$

• •

$m=100$,

• •

100 different $\beta$ values from 0.01 to 20, and

• •

30 different noise levels $\sigma$ from 0.1 to 10.

First we show the information content for different values of $k$, $\sigma$ and $\beta$. Since $d=10$ we look at $k\in\{1,2,...,5\}$ only. For $k\in\{6,7,...,10\}$ the behavior will be equivalent and the only difference is that the role of 0 and 1 ($\mu_{j}\in\{0,1\}$) is reversed.

For a given $k$ we see that, as in the non-sparse case, for low noise levels, increases in resolution obtain diminishing gains in information content while for higher noise levels, once the optimum resolution $\beta^{*}$ is surpassed, the information content decreases. Also notice that as $k$ increases toward $\frac{d}{2}=5$ the information content also increases. This is because the size of the hypothesis space is increasing in $k$ from 0 to $\lceil\frac{d}{2}\rceil$. We now show the generalization capacity for different values of $k$.

Again, since values of $k$ closer to $\lceil\frac{d}{2}\rceil$ correspond to larger hypothesis spaces, the generalization capacity of the cost function $\hat{R}(\mu,\xi)$ is larger for these $k$.

3.3 Sampling algorithm

As was mentioned in Section 1.2 we are ultimately interested in the sparse case where $k$ is kept constant and $d$ grows toward infinity. As $d$ grows, the size of the hypothesis space $\mathcal{C}^{k}$ grows exponentially fast which means computing the partition functions $Z_{\beta}(\xi)$ and $\Delta Z_{\beta}(\xi_{1},\xi_{2})$, which are sums over $\mathcal{C}^{k}$, quickly becomes unfeasible. In this section we use the sampling algorithm suggested in Buhm14 to estimate $Z_{\beta}(\xi)$ and $\Delta Z_{\beta}(\xi_{1},\xi_{2})$ without summing over the entire hypothesis space.

Recall that:

[TABLE]

If we let $p(\mu)=\frac{1}{|\mathcal{C}^{k}|}=\frac{1}{{d\choose k}}$ then,

[TABLE]

The last expression suggests the following sampling algorithm to estimate the generalization capacity $I$:

Choose a grid of relevant $\beta$ values: $\underline{\beta}=(\beta_{1},...,\beta_{l})$ 2. 2.

For $i=1$ to $m$

1. a.

Simulate $\xi_{1}^{i},\xi_{2}^{i}\sim N(0,I_{d})$ 2. b.

Uniformly sample $r$ hypotheses $\mu_{j}\in\mathcal{C}^{k}$ 3. c.

For $k=1$ to $l$

• •

Calculate quasi information content:

[TABLE] 3. 3.

For $k=1$ to $l$

• •

Estimate mean information content:

[TABLE]

• •

Estimate generalization capacity:

[TABLE]

3.4 Simulation results: sampling algorithm

We used the following parameters for the simulation experiments:

• •

$d=10$,

• •

$k=4$,

• •

$m=100$,

• •

$r=100,1000$,

• •

20 different $\beta$ values from 0.01 to 10, and

• •

20 different noise levels $\sigma$ from 0.1 to 15.

We first compare the generalization capacity estimation using the exhaustive and sampling algorithms for two different choices of $r$.

It is clear that using the sampling algorithm with $r=100$ we underestimate the generalization capacity since for very low noise levels we know the generalization capacity is equal to $\log|\mathcal{C}^{k}|=\log|{d\choose k}|=\log|{10\choose 4}|\approx 5.35$. For $r=1000$ the estimation is much better, however since $|\mathcal{C}^{k}|=210$ it is cheaper to use the exhaustive algorithm. Next we check the generalization capacity estimate for both algorithms as $d$ increases. The parameters used for the simulation experiments were the following:

• •

$d=5,6,...,20$,

• •

$k=4$,

• •

$m=100$,

• •

$r=20,100,500,1000$,

• •

20 different $\beta$ values from 0.01 to 10, and

• •

2 different noise levels $\sigma=2,4$.

The figure above confirms that the sampling algorithm proposed turns out to be more expensive than the exhaustive algorithm. For example, to estimate the generalization capacity accurately when $d=11$ we need $r=1000$ which is larger than the size of the hypothesis space ${11\choose 4}=330$. The sampling algorithm represents a way to estimate genearalization capacity by summing over a sample of the hypothesis class. The sample size required must be, at most, linear in $d$ if it is to be useful in estimating generalization capacity for sparse conditions when $k$ is fixed and $d$ is large.

3.5 Importance sampling algorithm

The aim of the sampling algorithm is to estimate $\mathbb{E}_{\mu}[e^{-\beta\hat{R}(\mu,\xi)}]$ with a sample from $p(\mu)=\frac{1}{|\mathcal{C}^{k}|}$ instead of exhaustively calculating it as:

[TABLE]

This means sampling $\mu_{j}\in\mathcal{C}^{k}$, calculating the Boltzmann weights $w_{\beta}(\mu_{j},\xi)$ and averaging them. In general, different hypothesis $\mu_{j}\in\mathcal{C}^{k}$ contribute differently to the average. The lower the cost of a hypothesis the larger the weight contributed. Those hypotheses which contribute most of the weight are relatively small in number. This means that with a small number of samples the proportion of important and unimportant hypotheses will not accurately reflect the population proportions and the result will be a biased estimate of $Z_{\beta}(\xi)$ and $\Delta Z_{\beta}(\xi_{1},\xi_{2})$. If we take a large sample, of size bigger than the size of $\mathcal{C}^{k}$, the simulation results seem to show that the estimates converge to those of the exhaustive algorithm, but this defeats the purpose of sampling hypotheses: to obtain an algorithm that is not exponential in $d$ as the size of the hypothesis class is.

We want to design an importance sampling algorithm where:

We sample according to a proposal distribution $q(\mu)$ which assigns more probability to more important hypotheses. 2. 2.

When estimating the expectation with the average we assign weights $w(\mu)$ to each sample to correct for the fact we sampled according to wrong distribution:

[TABLE]

Where $w(\mu)=\frac{p(\mu)}{q(\mu)}$.

Recall that we may write:

[TABLE]

Observe that $\mu^{T}1=k$ is constant for the case where the hypothesis space is $\mathcal{C}^{k}$ and not $\mathcal{C}=\mathbb{B}^{d}$. Also note that $h:=\mu^{T}\mu^{0}\in\{\max(0,2k-d),...,k\}$ is the number of hits of hypothesis $\mu$: the number of components $j$ such that $\mu_{j}=\mu_{j}^{0}=1$. With this in mind we can obtain the following equivalent expression for $\hat{R}(\mu,\xi)$:

[TABLE]

Where $h:=h(\mu,\mu^{0})=\mu^{T}\mu^{0}$. This means we may express the Boltzmann weights as:

[TABLE]

The importance of a hypothesis $\mu$ is given by the Boltzmann weights $w_{\beta}(\mu,\xi)$ and these depend on the functions $f_{\beta}$ and $g_{\beta}$. We first analyse $g_{\beta}(\mu,\xi):=e^{2\beta\frac{\sigma}{\sqrt{n}}\mu^{T}\xi}$. We have that, for $\beta,k>0$:

[TABLE]

This means that we can expect $g_{\beta}(\mu,\xi)$ to increase $w_{\beta}(\mu,\xi)$ for half the samples $\xi$ and decrease it for the other half, although in general, we can expect $g_{\beta}(\mu,\xi)$ to increase $w_{\beta}(\mu,\xi)$. So we see that $g_{\beta}(\mu,\xi)$ may contribute to a hypothesis being more or less important depending on the $\xi$ sampled. If we want to take this effect into account in determining the proposal distribution $q(\mu)$ we would have to make it depend on the given $\xi$ sampled. For simplicity we only take into account $f_{\beta}(h)$ in determining $q(\mu)$.

We now analyse $f_{\beta}(h)=e^{2\beta h}$. The more hits $h$ that a hypothesis $\mu$ has with respect to $\mu^{0}$ the lower the costs $\hat{R}(\mu,\xi)$ which means the larger he weight $w_{\beta}(\mu,\xi)$ contributed. Also note that there are ${k\choose h}{d-k\choose k-h}$ hypotheses $\mu\in\mathcal{C}^{k}$ such that $h(\mu,\mu^{0})=\mu^{T}\mu^{0}=h$. The following figure illustrates the importance $f_{\beta}(h)$ of a hypothesis $\mu\in\mathcal{C}^{k}$ and the number of such hypotheses in $\mathcal{C}^{k}$ as a function of $h$ for $d=20$, $k=4$ and $\beta=1$.

Hypothesis $\mu$ such that $h(\mu,\mu^{0})$ is close to $k$ contribute moste of the weight but are relatively few in number. There are ${k\choose h}{d-k\choose k-h}$ hypotheses $\mu\in\mathcal{C}^{k}$ such that $h(\mu,\mu^{0})=\mu^{T}\mu^{0}=h$ so there are $\sum_{h=\max(1,2k-d)}^{k}{k\choose h}{d-k\choose k-h}$ hypotheses such that $h>0$. As $d$ grows this becomes a low proportion of ${d\choose k}$, the total number of hypotheses. In other words, as $d$ grows, the most important hypotheses, those that contribute most of the weight, become a smaller proportion of all hypotheses.

This means that to sample a representative proportion of important hypotheses we need a very large sample. To find a way around this we use a proposal distribution $q(\mu)$ such that the probability of sampling a hypothesis with $h$ hits is the same for all $h$.

Let,

[TABLE]

This means that:

[TABLE]

[TABLE]

The above suggests the following importance sampling algorithm for estimating generalization capacity.

1. I.

Choose a grid of relevant $\beta$ values: $\underline{\beta}=(\beta_{1},...,\beta_{l})$ 2. II.

For $i=1$ to $m$

Simulate $\xi_{1}^{i},\xi_{2}^{i}\sim N(0,I_{d})$ 2. 2.

For $j=1$ to $r$

1. a.

Sample from $h_{j}\sim q(h)=\frac{1}{k-\max(0,2k-d)+1}$ 2. b.

Sample from $\mu_{j}|h_{j}\sim q(\mu|h)=\frac{\mathbbm{1}_{\{\mu^{T}\mu^{0}=h\}}}{{k\choose h}{d-k\choose k-h}}$ This can be done by:

1. i.

Calculate number of misses $m_{j}=k-h_{j}$ 2. ii.

Identify $A_{0}=\{a:\mu_{a}^{0}=0\}$ and $A_{1}=\{a:\mu_{a}^{0}=1\}$ 3. iii.

Set $\mu_{j}\leftarrow\mu^{0}$ 4. iv.

Uniformly sample $m_{j}$ times $r\in A_{0}$ and $s\in A_{1}$ setting $\mu_{jr}=1$ and $\mu_{js}=0$ each time. 3. 3.

For $k=1$ to $l$

• •

Calculate quasi information content:

[TABLE] 3. III.

For $k=1$ to $l$

• •

Estimate mean information content:

[TABLE]

Where $w_{i}=w(\mu_{i})=\frac{{k\choose\mu_{i}^{T}\mu^{0}}{d-k\choose k-\mu_{i}^{T}\mu^{0}}}{{d\choose k}}$

• •

Estimate generalization capacity:

[TABLE]

3.6 Simulation results: importance sampling algorithm

We used the following parameters for the simulation experiments:

• •

$d=10$,

• •

$k=4$,

• •

$m=100$,

• •

$r=100,1000$,

• •

20 different $\beta$ values from 0.01 to 10, and

• •

20 different noise levels $\sigma$ from 0.1 to 15.

We first compare the generalization capacity estimation using the exhaustive, sampling and importance sampling algorithms for two different choices of $r$.

The figures suggest that the generalization capacity estimated with the importance sampling algorithm converges to the exhaustive algorithm results for a sample size much smaller than is required for the sampling algorithm. Next we check the generalization capacity estimate for all three algorithms as $d$ increases. The parameters used for the simulation experiments were the following:

• •

$d=5,6,...,20$,

• •

$k=4$,

• •

$m=100$,

• •

$r=20,100,500,1000$,

• •

20 different $\beta$ values from 0.01 to 10, and

• •

2 different noise levels $\sigma=2,4$.

The above figures seem to confirm that the importance sampling algorithm need a much smaller sample size to converge than does the sampling algorithm suggesting it will be useful in estimating generalization capacity for large values of $d$ when we can no longer exhaustively evaluate the partition functions. In the following simulation experiments we estimate the generalization capacity of the cost function $\hat{R}(\mu,\xi)$ in sparse settings where $k$ is constant and $d$ is large. We used the following parameters for these simulation experiments:

• •

$d=10,...,10000$,

• •

$k=4$,

• •

$m=100$,

• •

$r=100$,

• •

100 different $\beta$ values from 0.01 to 30, and

• •

4 different noise levels $\sigma=3,4,5,6$.

Recall we ended Section 1.6 by mentioning that we can use generalization capacity to choose between alternative cost functions. The following simulation experiments, carried out with the above parameters, were performed for two different cost functions, the squared-loss based risk function and an absolute-loss based risk function. Recall expression 2.1.0.8 for the squared-loss based risk function:

[TABLE]

If we replace the L2 norm with the L1 norm we get te absolute-loss based risk function:

[TABLE]

The following figure shows the estimated generalization capacity of both cost functions for different number of components $d$ and noise levels $\sigma$. Each scatter point represents the estimation, by simulation, of the generalization capacity for a given $d$ and cost function $\hat{R}(\mu,\xi)$. Blue points represent the generalization capacity of the L1 norm cost function and black points that of the L2 norm cost function. The blue and black lines are smoothing splines applied to the blue and black points respectively.

We can see that the generalization capacity for both cost functions, as a function of the number of components $d$, displays concave behavior. Initially, as the hypothesis set size grows we see rapid increase in the generalization capacity however after a certain threshold, the large number of hypotheses makes detection of the relevant components difficult so that additional gains in generalization capacity are marginal. The value of this threshold depends on the noise level of the data: the higher the noise level the lower the threshold. Additionally, observe that, as we would expect given that the data has a gaussian distribution, the L2 norm cost function has a greater generalization capacity than that of the L1 norm cost function.

Chapter 4 Towards sparse feature selection for correlated features

As with the model described in Section 1.2 we deal with the statistical model:

[TABLE]

Except in this case we have that:

• •

$||\mu^{0}||_{1}=k$,

• •

$\mu^{0}\in\mathbb{B}^{d}_{k}=\{\mu\in\mathbb{B}^{d}:||\mu||_{1}=k\}$,

• •

$|\mathbb{B}^{d}_{k}|={d\choose k}$,

• •

It is assumed that $k$ is known,

• •

$\epsilon\sim N(0,\Gamma)$ where $\Gamma_{ii}=\sigma^{2}$ for all $i\in\{1,...,d\}$, and

• •

observations $X_{i}$ with $i=1,...,n$ are i.i.d.

Notice that in contrast to the problem statement in section 1.2, the features are now correlated. This is because each entry represents a variable or feature and the normal situation is that features are correlated. We now change the sparsity condition from a small, fixed $k$ and large $d$ to $k\approx\frac{d}{\log(d)}$. This reflects the assumption that the number of relevant features grows as the number of total features grows, albeit at a slower rate.

We are ultimately interested in estimating, by simulation, the generalization capacity of the quadratic loss based, empirical risk function. We could try to use the importance sampling algorithm of Section 3.5 although we haven’t tested for the case where $k\approx\frac{d}{\log d}$ and $\Sigma\neq I_{d}$. In this Chapter we explore an alternate way to approximate the expected value of the log partition function $\mathbb{E}_{X}[\log Z_{\beta}(X)]$, which, as we have seen, is a component of the generalization capacity. The following derivations and approximations are based on BuhmNotes.

In this case we have that

[TABLE]

So we let $\xi=\frac{\bar{X}-\mu^{0}}{\nicefrac{{\sigma}}{{\sqrt{n}}}}\sim N(0,\Sigma)$, where $\Sigma=\frac{\Gamma}{\sigma^{2}}$ so that $\bar{X}=\mu^{0}+\frac{\sigma}{\sqrt{n}}\xi$. Recall from 2.5.0.26 that we may write the square based empirical risk function as:

[TABLE]

Let $\hat{r}(\xi)=1-2\frac{\sigma}{\sqrt{n}}\xi-2\mu^{0}$ and $\mathcal{C}^{k}_{h}=\{\mu\in\mathcal{C}^{k}:\mu^{T}\mu^{0}=h\}$. Then:

[TABLE]

with which

[TABLE]

but since $\mathcal{C}^{k}_{k}=\{\mu^{0}\}$ we have that

[TABLE]

Let

[TABLE]

Notice that $\bar{B}_{k}=1$ so that

[TABLE]

If we take a hypothesis $\mu$ at random the number of hits $h:=h(\mu)$ is a hypergeometric random variable:

[TABLE]

Where

• •

$H$ measures the number of succesesses from $n$ draws without replacement,

• •

$S$ is the number of success states and

• •

$n$ is the number of draws.

In this case we pick a $\mu$ at random which represents picking $k$ components to be equal to one (without replacement), from a population of $d$ where there are exactly $k$ components equal to one. This random variable has the following probability distribution function:

[TABLE]

Where $h\in\{\max(0,n+S-N),...,\min(n,S)\}=\{\max(0,2k-d),...,k\}$ which means that

[TABLE]

Now $|\mathcal{C}_{h}^{k}|$ is exponential in $d$ and $\bar{B}_{h}$ is a sum over a very large hypothesis space so we will need a way to approximate it. We have that:

[TABLE]

where $\eta(\mu):=2\frac{\sigma}{\sqrt{n}}\mu^{T}\xi$. First notice that if $\mu=\mu^{0}$ then $h=k$, $\mu^{T}\hat{r}(\xi)=-k-\eta(\mu)$ and since $\xi\sim N(0,\Sigma)$ we have that \mu^{T}\mathbb{E}_{\xi}\bigg{[}\hat{r}(\xi)\bigg{]}=-k so that:

[TABLE]

Where $Y_{h}:=\sum_{\mu\in\mathcal{C}^{k}_{h}}e^{\beta\eta(\mu)}$ . Since

[TABLE]

so that

[TABLE]

The problem with evaluating $Y_{h}$ is that it is a sum over $\mathcal{C}^{k}_{h}$ which is very large if $d$ is large and $k\approx\frac{d}{\log d}$. If we can come up with a good approximation $\widetilde{Y}_{h}$ of $Y_{h}$ then the above formula suggests the following algorithm for estimating \mathbb{E}_{\xi}\bigg{[}\log Z_{\beta}(\xi)\bigg{]}, for a given $\beta$, $d$ and $k$:

For $i=1$ to $m$

1. a.

Simulate $\xi^{i}\sim N(0,\Sigma)$ 2. b.

For $j=1$ to $p$

• •

Simulate $H_{j}\sim Hypergeometric(k,d,k)$

• •

Calculate $a_{j}$

[TABLE] 3. c.

Calculate $\bar{a}^{i}=\frac{1}{p}\sum_{j=1}^{p}a_{j}$ 2. 2.

Estimate \mathbb{E}_{\xi}\bigg{[}\log Z_{\beta}(\xi)\bigg{]}

[TABLE]

In order to explore possible ways to approximate $Y_{h}$ we take a look at its distribution.

[TABLE]

Since

[TABLE]

and

[TABLE]

we have that even in the case that $\Sigma=I_{d}$, $Y_{h}(\xi)$ is a sum of $|\mathcal{C}^{k}_{h}|$ correlated log-normals with correlation matrix $\Lambda$ such that

[TABLE]

If $\Sigma=I_{d}$ we have that

[TABLE]

and

[TABLE]

Where $c_{ij}:=\mu_{i}^{T}\mu_{j}$ is the number of overlapping ones in $\mu_{i}$ and $\mu_{j}$. If we can use the approximation proposed in BuhmNotes where we assume $\eta(\mu)\approx\eta_{h}$ we have that:

[TABLE]

However it is not clear that this approximation will be helpful since it corresponds to assuming that if $\mu_{i},\mu_{j}\in\mathcal{C}^{k}_{h1}$ and $\mu_{p}\in\mathcal{C}^{k}_{h2}$ then $Cor(\beta\eta(\mu_{i}),\beta\eta(\mu_{j}))=1$ and $Cor(\beta\eta(\mu_{i}),\beta\eta(\mu_{p}))=0$ for all $i,j\in\{1,...,{k\choose h1}{d-k\choose k-h1}\}$ and $p\in\{1,...,{k\choose h2}{d-k\choose k-h2}\}$.

Chapter 5 Summary

The work presented in this report falls into the following three categories:

1. i.)

Exposition of the theory related to approximation set coding and generalization capacity: Sections 1.3-1.6, 2. ii.)

Implementation of the generalization capacity concept to the sparse mean localization: Section 1.2 and Chapters 2-3, and 3. iii.)

Exploration of how generalization capacity can be implemented for a more realistic version of the sparse feature selection problem: Chapter 4.

In the first case we introduce pattern analysis which is a broad framework with which to deal with learning problems. We describe the approximation set approach to learning where we look for a set of hypotheses with similar performance instead of looking for just one. With the definitions from pattern analysis and approximation sets we then introduce information theoretic concepts, originally developed within the field of statistical physics, which we need to define GC. To motivate the meaning of GC we study Shannon’s noisy channel coding theorem and related theory drawing an extensive analogy between channel capacity and generalization capacity. We describe in detail the differences and similarities between the communication scenario from which Shannon’s concept of channel capacity arises and the learning scenario from which Buhm13’s concept of generalization capacity can be derived.

The second type of contribution presented here involves the implementation of generalization capacity to the problem of sparse mean localization. In Buhm14 an expression for GC for the sparse mean localization problem (with respect to the squared loss based empirical risk function) is derived that suggests a simulating algorithm for its estimation. The simulating algorithm is briefly described and results are presented. Based on this work we designed and detailed successive simulating algorithms for GC estimation for non-sparse and sparse mean localization problems. We first designed an exhaustive sampling algorithm for non-sparse mean localization in low dimensional cases where the dimension $d$ of the mean vector $\mu^{0}$ is in the order of 10. We implemented various versions of common random numbers to reduce variance and chose the best one. We then applied it to the sparse mean localization problem and tried to scale it up to deal with high dimensional cases, but found that since the size of the hypothesis class is exponential in the dimension $d$ the exhaustive algorithm is unfeasible since it involves summing over the entire hypothesis space. To deal with this we designed a uniform sampling algorithm, also suggested in Buhm14, but found that for this algorithm to converge the number of sampled hypotheses necessary was actually larger than the hypothesis class size. We solved this difficulty by designing an importance sampling alogorithm which seems to be adequate for high dimensions in the order of 10,000 (asymptotic confidence intrevals need to be derived to verify this). In Buhm14 results showing the estimation of GC for high dimensional sparse mean localization are shown however the simulation method is not detailed. The contribution of this work is to detail and justify numerical estimation of GC for the sparse mean localization problem. It is also worth mentioning that the results of the estimatation of GC are qualitatively different to those found in Buhm14:compare for example figures 1 and 2 in Buhm14 to figure 3.10.

Finally, based on BuhmNotes, we explored estimating GC in a more realistic version of the sparse feature selection problem and sketched a general simulation algorithm, describing certain difficulties that need to be resolved.

5.1 Future work

Possible ways to extend the work presented here are:

• •

Derive asymptotic confidence intervals for the different GC estimators (exhaustive, sampling, importance sampling) so as to verify when these are fullfilled and so determine the number of simulations necessary. This is especially important for the sparse high dimensional cases ($d$ large) so as to establish the reliability of the estimates produced and the feasability of the algorithms.

• •

Explore the sparse feature selection problem further: does the $\eta(\mu)\approx\eta_{h}$ approximation work? What other approximations can we use to make the estimation of $Y_{h}(\xi)$ feasible for large $d$ and $k\approx\frac{d}{\log d}$.

• •

Implement the importance sampling algorithm to the more general sparse feature selection problem and compare with simulation methods based on approximations of the type discussed in Chapter 4.

• •

Extend the study to include estimation of sparse histograms. In this case a histogram can be represented as a vector $X_{i}$ composed of a mean $\mu^{0}$ and noise $\epsilon$: i.e. $X_{i}=\mu^{0}+\epsilon_{i}$ as in the sparse mean localization problem. However in this case $\mu^{0}\in[0,1]^{d}$ instead of $\mu^{0}\in\{0,1\}^{d}$ and $|\mu^{0}|_{1}=1$ instead of $|\mu^{0}|_{1}=k$. The number of non-zero entries is still assumed to be $k$ to enforce a sparsity condition.

\phantomsection

Appendix A R Code

We include the code in R that was used to implement the importance sampling algorithm. Other algorithms described to estimate generalization capacity are very similar so we omit them.

A.1 Functions

A.2 Script