Optimal stimulus and noise distributions for information transmission via suprathreshold stochastic resonance

TL;DR

This paper investigates how noise can enhance information transmission in neural-like systems through suprathreshold stochastic resonance, deriving optimal noise and stimulus distributions that maximize channel capacity.

Contribution

It provides a theoretical framework linking mutual information and Fisher information to identify optimal stimulus and noise distributions for SSR.

Findings

Capacity is maximized when the signal distribution matches Jeffrey's prior.

Optimal noise distribution relates to the signal distribution via a cosine function.

Theoretical results support previous empirical and computational studies.

Abstract

Suprathreshold stochastic resonance (SSR) is a form of noise enhanced signal transmission that occurs in a parallel array of independently noisy identical threshold nonlinearities, including model neurons. Unlike most forms of stochastic resonance, the output response to suprathreshold random input signals of arbitrary magnitude is improved by the presence of even small amounts of noise. In this paper the information transmission performance of SSR in the limit of a large array size is considered. Using a relationship between Shannon's mutual information and Fisher information, a sufficient condition for optimality, i.e. channel capacity, is derived. It is shown that capacity is achieved when the signal distribution is Jeffrey's prior, as formed from the noise distribution, or when the noise distribution depends on the signal distribution via a cosine relationship. These results provide theoretical verification and justification for previous work in both computational neuroscience and electronics.

Figures

Tables

Table 1. Table 1: The auxiliary PDF, f Q ​ ( τ ) subscript 𝑓 𝑄 𝜏 f_{Q}(\tau) , for five different ‘matched’ signal and noise distributions (i.e. same distribution but with different variances), as well as H ​ ( Q ) 𝐻 𝑄 H(Q) , the entropy of f Q ​ ( τ ) subscript 𝑓 𝑄 𝜏 f_{Q}(\tau) . The threshold value, θ 𝜃 \theta , and the signal and noise means are assumed to be zero, so that these results are independent of θ 𝜃 \theta . The noise intensity, σ = σ η / σ x 𝜎 subscript 𝜎 𝜂 subscript 𝜎 𝑥 \sigma=\sigma_{\eta}/\sigma_{x} , is the ratio of the noise standard deviation to the signal standard deviation. For the Cauchy case, σ λ subscript 𝜎 𝜆 \sigma_{\lambda} is the ratio of the full-width-at-half-maximum parameters. The label ‘NAS’ indicates that there is no analytical solution for the entropy.

Distribution	$f_Q\left(\tau \right)$	$H\left(Q\right)$
Gaussian	$\sigma \exp \left(\left(1-{\sigma }^2\right){\left({\text{erf}}^{-1}\left(2\tau -1\right)\right)}^2\right)$	$-{\log }_2\left(\sigma \right)-\frac{1}{2\ln 2}\left(\frac{1}{{\sigma }^2}-1\right)$
Uniform, $\sigma \ge 1$		${\log }_2\sigma$
Laplacian		$-{\log }_2\left(\sigma \right)-\frac{1}{2\ln 2}\left(\frac{1}{\sigma }-1\right)$
Logistic	$\sigma \frac{{\left(\tau \left(1-\tau \right)\right)}^{\left(\sigma -1\right)}}{{\left({\tau }^{\sigma }+{\left(1-\tau \right)}^{\sigma }\right)}^2}$	NAS
Cauchy	${\sigma }_{\lambda }\frac{1+{\tan }^2\left(\pi \left(\tau -0.5\right)\right)}{\left(1+{\sigma }_{\lambda }^2{\tan }^2\left(\pi \left(\tau -0.5\right)\right)\right)}$	NAS

Table 2. Table 2: Large N 𝑁 N channel capacity and optimal σ 𝜎 \sigma for ‘matched’ signal and noise

Distribution	$C(X,y)-0.5{\log }_2\left(N\right)$	${\sigma }_o$	$C(X,y)-{I(X,y)|}_{\sigma =1}$
Gaussian	$-0.3964$	$0.6563$	$0.208$
Logistic	$-0.3996$	$0.5943$	$0.205$
Laplacian	$-0.3990$	$0.5384$	$0.205$

Equations

$$D(f_X||f_Y)={\int }_{\eta \in S}{f}_X(\eta ){\log }_2\left(\frac{f_X(\eta )}{f_Y(\eta )}\right)d\eta .$$

$$I(X,Y)$$

$$=H(Y)-H(Y|X)\text{mbox}bitspersample.$$

$$H(Y)=-\sum _{n=0}^N{P}_Y(n){\log }_2{P}_Y(n),$$

$$H(X)=-{\int }_{\eta \in S}{f}_X(\eta ){\log }_2(f_X(\eta ))d\eta .$$

$$H(Y|X)=-{\int }_{x\in S}{f}_X(x)\sum _{n=0}^N{P}_{Y|X}(n|x){\log }_2(P_{Y|X}(n|x))dx.$$

$$\text{mbox}Find:C=\underset{\{f_X(x)\}}{\max }I(X,Y).$$

$$P_y(n)={\int }_x{P}_{y|X}(n\mid x)f_X(x)dxn\in 0,..,N.$$

$$I(X,y)=$$

$$=$$

$$\left(-{\int }_{-\infty }^{\infty }{f}_X(x)\sum _{n=0}^N{P}_{y|X}(n\mid x){\log }_2{P}_{y|X}(n\mid x)dx\right).$$

$$P_{1\mid x}=1-F_{\eta }(\theta -x).$$

$$P_{y|X}(n|x)=(\genfrac{}{}{0ex}{}{N}{n})P_{1|x}^n(1-P_{1|x}{)}^{N-n}n\in 0,..,N,$$

$$I(X,y)=$$

$$N{\int }_x{f}_X(x)P_{1\mid x}{\log }_2{P}_{1\mid x}dx+$$

$$N{\int }_x{f}_X(x)(1-P_{1\mid x}){\log }_2(1-P_{1\mid x})dx.$$

$$I(X,y)={\log }_2(N+1)-\frac{N}{2\ln 2}-\frac{1}{N+1}\sum _{n=2}^N(N+1-2n){\log }_{2}n.$$

$$f_{\bar{Y}}(\bar{y})=$$

$$=$$

$$f_{Q}(\tau)=\frac{f_{X}(x)}{f_{\eta}(\theta-x)}\Big{|}_{x=\theta-F_{\eta}^{-1}(1-\tau)},\quad\tau\in[0,1],$$

$$f_{\bar{Y}}(\bar{y})=\frac{f_Q\left(\frac{\bar{y}}{N}\right)}{N},$$

$$I(X,y)=$$

$$N{\int }_{\tau =0}^{\tau =1}{f}_Q(\tau )\tau {\log }_2\tau d\tau +$$

$$N{\int }_{\tau =0}^{\tau =1}{f}_Q(\tau )(1-\tau ){\log }_2(1-\tau )d\tau ,$$

$$P_y(n)=\left(\genfrac{}{}{0ex}{}{N}{n}\right){\int }_{\tau =0}^{\tau =1}{f}_Q(\tau ){\tau }^n(1-\tau {)}^{N-n}d\tau .$$

$$H(Q)$$

$$=-{\int }_x{f}_X(x){\log }_2\left(\frac{f_X(x)}{f_{\eta }(\theta -x)}\right)dx$$

$$=-D(f_X(x)||f_{\eta }(\theta -x)),$$

$$J(x)={\left(\frac{d{P}_{1|x}}{dx}\right)}^2\frac{N}{P_{1|x}(1-P_{1|x})}.$$

$$I(X,y)$$

$$H(y)$$

$$I(X,y)\simeq 0.5{\log }_2\left(\frac{N\pi }{2e}\right)-D(f_X||f_S),$$

$$f_S(x)=\frac{\sqrt{J(x)}}{\pi \sqrt{N}}.$$

$$f_S(x)=\frac{f_{\eta }(\theta -x)}{\pi \sqrt{F_{\eta }(\theta -x)(1-F_{\eta }(\theta -x))}}.$$

$$f_X(x)=f_S(x)\forall x,$$

Full text

I Introduction

The term ‘stochastic resonance’ describes the situation where a system’s response to some signal is optimized by the presence of random noise, rather than its absence. It occurs in a wide variety of nonlinear physical Gammaitoni et al. (1998) and biological Moss et al. (2004) systems.

In many of the systems and models in which stochastic resonance (SR) has been observed, the essential nonlinearity is a single static threshold, e.g. Wiesenfeld et al. (1994); Bulsara and Zador (1996); Chapeau-Blondeau (1996); Greenwood et al. (1999). It is generally thought that SR cannot occur in such systems for suprathreshold signals, meaning that the amplitude of the input signal needs to be restricted to values smaller than the amplitude of the threshold for SR to occur DeWeese and Bialek (1995).

However, the 1999 discovery of a novel form of SR—known as suprathreshold stochastic resonance (SSR)—showed that this is not always true Stocks (2000). SSR occurs in an array of identical threshold nonlinearities, each of which are subject to independently random additive noise. We refer to this array as the SSR model—see Fig. 1. In this model SR occurs regardless of whether the input signal is entirely subthreshold or not. Furthermore, SSR occurs even for very large input SNRs. This is a further difference to conventional SR, for which the signal is required to be weak compared to the noise.

SSR is a form of aperiodic stochastic resonance Collins et al. (1995); Bulsara and Zador (1996); Morse and Roper (2000) that was first shown to occur by calculating Shannon’s average mutual information for the SSR model Stocks (2000). It was subsequently found that the performance achievable via SSR is maximized when all threshold values are set to the signal mean Stocks (2001a), and that for sufficiently small input SNRs, modifying the thresholds in the model cannot improve information transfer McDonnell et al. (2006).

The SSR model was originally motivated as a model for parallel sensory neurons, such as those synapsing with hair cells in the inner ear Stocks et al. (2002). Although the basic SSR model is non-dynamical, and does not model the many complexities of real neurons, each threshold nonlinearity is equivalent to a Pitts-McCulloch neuron model, and encapsulates the neural coding properties we are interested in—i.e. the generation of action potentials in response to a noisy aperiodic random stimulus. The small input SNRs we focus on are biologically relevant Bialek et al. (1993), particularly so for hair cells, which are subject to substantial Brownian motion Lindner et al. (2005). This leads to much randomness in the release of neurotransmitters at synapses with afferent neurons leading to the cochlear nucleus.

Further justification of the SSR model’s relevance to neural coding is discussed in Hoch et al. (2003); Durrant and Feng (2007); Nikitin et al. (2007), and by extensions of the model to include more biologically realistic neural features. For example, the parallel array has been modified to consist of parallel FitzHugh-Nagumo neuron models Stocks and Mannella (2001), leaky integrate-and-fire neuron models Hoch et al. (2003); Durrant and Feng (2007) and Hodgkin-Huxley models Hoch et al. (2003), and for the case of signal-dependent (multiplicative) noise Nikitin et al. (2007). In all cases the same qualitative results as for the simple threshold model were obtained. The SSR effect has also led to a proposal for improving the performance of cochlear implants for suprathreshold stimuli Stocks et al. (2002), based on the idea that the natural randomness present in functioning cochlear hair cells is missing in patients requiring implants Morse et al. (2007).

The purpose of this paper is to analyze, in a general manner, the information theoretic upper limits of performance of the SSR model. This requires allowing the array size, $N$, to approach infinity. Previous work has discussed the scaling of the mutual information through the SSR model with $N$ for specific cases, and found conditions for which the maximum mutual information—i.e. channel capacity—occurs Stocks (2001b, a); Hoch et al. (2003). In a neural coding context, the question of ‘what is the optimal stimulus distribution?’ for a given noise distribution is discussed numerically for the SSR model in Hoch et al. (2003).

In Sec. II, we significantly extend the results in Stocks (2001b, a); Hoch et al. (2003), by showing that the mutual information and output entropy can both be written in terms of simple relative entropy expressions—see Eqs. (21) and (22). This leads to a very general sufficient condition, Eq. (25), for achieving capacity in the large $N$ regime that can be achieved either by optimizing the signal distribution for a given noise distribution, or optimizing the noise for a given signal. Given the neuroscience motivation for studying the SSR model, this result is potentially highly significant in computational neuroscience, where both optimal stimulus distributions, and optimal tuning curves are often considered Hoch et al. (2003); Brunel and Nadal (1998).

Furthermore, the optimal signal for the special case of uniform noise is shown to be the arcsine distribution (a special case of the Beta distribution), which has a relatively large variance and is bimodal. This result provides theoretical justification for a proposed heuristic method for analog-to-digital conversion based on the SSR model Nguyen (2007). In this method, the input signal is transformed so that it has a large variance and is bimodal.

As a means of verification of our theory, in Sec. III our general results are compared to the specific capacity results contained in Stocks (2001b, a); Hoch et al. (2003). This leads us to find and justify improvements to these previous results.

Before we proceed however, the remainder of this section outlines our notation, describes the SSR model, and derives some important results that we utilize.

I.1 Information Theoretic Definitions

Recent work using the SSR model has described performance using measures other than mutual information McDonnell et al. (2002a, b); Rousseau et al. (2003); Rousseau and Chapeau-Blondeau (2004); McDonnell et al. (2005); Wang and Wu (2005). However, in line with much theoretical neuroscience research Bialek et al. (1993), here we use the information theoretic viewpoint where the SSR model can be considered to be a communications channel Stocks (2000).

Throughout, we denote the probability mass function (PMF) of a discrete random variable, $\alpha$, as $P_{\alpha}(\cdot)$, the probability density function (PDF) of a continuous random variable, $\beta$, as $f_{\beta}(\cdot)$, and the cumulative distribution function (CDF) of $\beta$ as $F_{\beta}(\cdot)$.

All signals are discrete-time memoryless sequences of samples drawn from the same stationary probability distribution. This differs from the detection scenario often considered in SR research, in which the input signal is periodic. Such a signal does not convey new information with an increasing number of samples, and cannot be considered from an information theoretic viewpoint DeWeese and Bialek (1995).

Consider two continuous random variables, $X$ and $Y$, with PDFs $f_{X}(x)$ and $f_{Y}(x)$, with the same support, $S$. The relative entropy—or Kullback-Liebler divergence—between the two distributions is defined as Cover and Thomas (1991)

[TABLE]

Suppose $X$ and $Y$ have joint PDF, $f_{XY}(x,y)$. Shannon’s mutual information between $X$ and $Y$ is defined as the relative entropy between the joint PDF and the product of the marginal PDFs Cover and Thomas (1991),

[TABLE]

where $H(Y)$ is the entropy of $Y$ and $H(Y|X)$ is the average conditional entropy of $Y$ given $X$.

The definition of mutual information also holds for discrete random variables, and for one variable discrete and one continuous. The entropy of a discrete random variable, $Y$, is given by

[TABLE]

while a continuous random variable, $X$, has differential entropy

[TABLE]

In this paper we are interested in the case of $X$ continuous with support $S$ and $Y$ discrete, with $N$ states, in which case the average conditional entropy of $Y$ given $X$ is

[TABLE]

In information theory, the term channel capacity is defined as being the maximum achievable mutual information of a given channel Cover and Thomas (1991). Suppose $X$ is the source random variable, and $Y$ is the random variable at the output of the channel. Usually, the channel is assumed to be fixed and the maximization performed over all possible source PDFs, $f_{X}(x)$. The channel capacity, $C$, can be expressed as the optimization problem,

[TABLE]

Usually there are prescribed constraints on the source distribution such as a fixed average power, or a finite alphabet Cover and Thomas (1991). In Sec. III we will also consider the more stringent constraint that the PDF of the source is known other than its variance. In this situation, channel capacity is determined by finding the optimal source variance, or as is often carried out in SR research, the optimal noise variance.

I.2 SSR Model

Fig. 1 shows a schematic diagram of the SSR model. The array consists of $N$ parallel threshold nonlinearities—or ‘devices’, each of which receive the same random input signal, $X$, with PDF $f_{X}(\cdot)$. The $i$–th device in the model is subject to continuously valued iid—independent and identically distributed—additive random noise, $\eta_{i}$ ($i=1,..,N$), with PDF $f_{\eta}(\cdot)$. Each noise signal is required to also be independent of the signal, $X$. The output of each device, $y_{i}$, is unity if the input signal, $X$, plus the noise on that device’s threshold, $\eta_{i}$, is greater than the threshold value, $\theta$. The output signal is zero otherwise. The outputs from each device, $y_{i}$, are summed to give the overall output signal, $y=\sum_{i=1}^{N}y_{i}$. This output is integer valued, $y\in[0,..,N]$, and is therefore a quantization (digitization) of $X$ McDonnell et al. (2005).

The conditional PMF of the output given the input is $P_{y|X}(y=n|X=x),n\in[0..,N]$. We abbreviate this to $P_{y|X}(n|x)$. The output distribution is

[TABLE]

The mutual information between $X$ and $y$ is that of a semi-continuous channel Stocks (2000), and can be written as

[TABLE]

To progress further we use the notation introduced in Stocks (2000). Let $P_{1{\mid}x}$ be the probability of the $i$–th threshold device giving output $y_{i}=1$ in response to input signal value, $X=x$. If the noise CDF is $F_{\eta}(\cdot)$, then

[TABLE]

As noted in Stocks (2000), $P_{y|X}(n|x)$ is given by the binomial distribution as

[TABLE]

and Eq. (I.2) reduces to

[TABLE]

Numerically evaluating Eq. (I.2) as a function of input SNR for given signal and noise distributions finds that the mutual information has a unimodal stochastic resonance curve for $N>1$, even when the signal and noise are both suprathreshold—i.e. the threshold value, $\theta$, is set to the signal mean Stocks (2001a); McDonnell et al. (2002a).

Further analytical simplification of Eq. (I.2) is possible in the case where the signal and noise PDFs are identical with the same variance, i.e. $f_{X}(x)=f_{\eta}(\theta-x)$ $\forall~{}x$ Stocks (2001a). The result is

[TABLE]

What is quite remarkable about this result is that the mutual information is independent of the shape of the PDFs of the signal and noise, other than that $f_{X}(x)=f_{\eta}(\theta-x)~{}\forall~{}x$. This means that both PDFs have the same shape, but may possibly have different means, and be mutually reversed along the $x$-axis about their means. In Sec. II.4 we compare the mutual information of Eq. (12) with our calculations of the general channel capacity.

I.3 Describing SSR Using a Single PDF, $f_{Q}(\tau)$

We now show that the mutual information in the SSR model depends solely on $N$, and an auxiliary PDF, $f_{Q}(\cdot)$. This PDF is shown to be that of the random variable describing the conditional average output of the SSR model, given that the input signal is $X=x$.

I.3.1 $f_{Q}(\tau)$ as the PDF of the Average Transfer

Function

Although the output of the SSR model, $y$, is a discrete random variable, the conditional expected value of $y$, given the input is $X=x$, is a continuous random variable, since $X$ is. We label this random variable as $\bar{Y}$. Since the PMF of $y$ given $X=x$ is the binomial PMF as in Eq. (10), we know that $\bar{Y}$ is the random variable that results from $\bar{y}=\mbox{E}[y|X=x]=NP_{1|x}$. Inverting this gives $x=\theta-F_{\eta}^{-1}\left(1-\frac{\bar{y}}{N}\right)$.

The PDF of $\bar{Y}$ can be derived from $f_{X}(\cdot)$, since $\bar{y}=NP_{1|x}$ provides an invertible transformation of $X$, with PDF $f_{X}(x)$, to $\bar{Y}$, with PDF $f_{\bar{Y}}(\bar{y})$. Using the well known expression for the resultant PDF, and provided the support of $f_{X}(x)$ is contained in the support of $f_{\eta}(\theta-x)$—since otherwise $\frac{dx}{d\bar{y}}$ does not necessarily exist—we have

[TABLE]

Our condition regarding the supports of the signal and noise ensures that $f_{\eta}(\cdot)\neq 0$. If we make a further change to a new random variable, $Q$, via $\tau=\frac{\bar{y}}{N}$, the PDF of $Q$ is

[TABLE]

and the PDF of $\bar{Y}$ can be written as

[TABLE]

which illustrates the physical significance of the auxiliary PDF, $f_{Q}(\cdot)$, as the PDF of $\frac{\bar{y}}{N}$.

I.3.2 Mutual Information in Terms of $f_{Q}(\tau)$

Making a change of variable in Eq. (I.2) from $x$ to $\tau$, via $\tau=P_{1|x}=1-F_{\eta}(\theta-x)$ gives

[TABLE]

where

[TABLE]

Eqs. (I.3.2) and (17) show that the PDF $f_{Q}(\tau)$ encapsulates the behavior of the mutual information in the SSR model.

I.3.3 Entropy of the random variable, $Q$

If we make a change of variable from $\tau$ to $x$, and note that $f_{X}(x)dx=f_{Q}(\tau)d\tau$, the entropy of $Q$ can be written as

[TABLE]

which is the negative of the relative entropy between the signal PDF, and the noise PDF reversed about $x=0$ and shifted by $\theta$. In the event that the noise PDF is an even function about its mean, and $\theta$ is equal to the signal mean, then the entropy of $Q$ is simply the negative of the relative entropy between the signal and noise PDFs.

I.3.4 Examples of the PDF $f_{Q}(\tau)$

The PDF $f_{Q}(\tau)$ can be derived for specific signal and noise distributions. Table 1 lists $f_{Q}(\tau)$ for several cases where the signal and noise share the same distribution and a mean of zero, but with not necessarily equal variances. The threshold value, $\theta$, is also set to zero.

For each case considered, the standard deviation of the noise can be written as $a\sigma_{\eta}$, where $a$ is a positive constant, and the standard deviation of the signal can be written $a\sigma_{x}$. We find that $f_{Q}(\tau)$ in each case is a function of a single parameter that we call the noise intensity, $\sigma=\sigma_{\eta}/\sigma_{x}$. Given this, from Eq. (I.3.2), it is clear that the mutual information must be a function only of the ratio, $\sigma$, so that it is invariant to a change in $\sigma_{x}$ provided $\sigma_{\eta}$ changes by the same proportion. This fact is noted to be true for the Gaussian case in Stocks (2000), and the uniform case in Stocks (2001a), but here we have illustrated why.

We note however, that if $\theta$ is not equal to the signal mean, then $f_{Q}(\tau)$ will depend on the ratio $\frac{\theta}{\sigma_{x}}$, as well as $\theta$ and $\sigma$, and therefore so will the mutual information.

Table 1 also lists the entropy of $Q$ for three cases where an analytical expression could be found.

I.4 Large $N$ SSR: Literature Review and Outline of This Paper

In the absence of noise, the maximum mutual information is the maximum entropy of the output signal, $\log_{2}{(N+1)}$. It has been shown for very specific signal and noise distributions that the mutual information in the SSR model scales with $0.5\log_{2}{(N)}$ for large $N$ Stocks (2001b, a). This means that the channel capacity for large $N$ under the specified conditions is about half the maximum noiseless channel capacity. This situation is discussed in Sec. III.

The only other work to consider SSR in the large $N$ regime finds that the optimal noise intensity for Gaussian signal and noise occurs for $\sigma\simeq 0.6$ Hoch et al. (2003). Unlike Stocks (2001b)—which uses the exact expression of Eq. (12), and derives a large $N$ expression by approximating the summation with an integral—Hoch et al. (2003) begins by using a Fisher information based approximation to the mutual information.

In Appendix A.1 we re-derive the formula of Hoch et al. (2003) in a different manner, which results in new large $N$ approximations for the output entropy, as well as the mutual information. These approximations provide the basis for the central result of this paper, which is a general sufficient condition for achieving channel capacity in the SSR model, for any arbitrary specified signal or noise distribution. This is discussed in Section II. These new general results are compared with the specific results of Stocks (2001b, a); Hoch et al. (2003) in Sec. III.

II A General Expression for the SSR Channel Capacity for Large $N$

Fisher information Lehmann and Casella (1998); Cover and Thomas (1991) has previously been discussed in numerous papers on both neural coding Bethge et al. (2002) and stochastic resonance Greenwood et al. (2004), and both Stemmler (1996); Greenwood et al. (2000). However, most SR studies using Fisher information consider only the case where the signal itself is not a random variable. When it is a random variable, it is possible to connect Fisher information and Shannon mutual information under special conditions, as discussed in Stemmler (1996); Brunel and Nadal (1998); Kang and Sompolinsky (2001); Hoch et al. (2003).

It is demonstrated in Hoch et al. (2003) that the Fisher information at the output of the SSR model as a function of input signal value $X=x$, is given by

[TABLE]

In Hoch et al. (2003), Eq. (19) is used to approximate the large $N$ mutual information in the SSR model via the formula

[TABLE]

This expression—which is derived under much more general circumstance in Clarke and Barron (1990); Brunel and Nadal (1998)—relies on an assumption that an efficient Gaussian estimator for $x$ can be found from the output of the channel, in the limit of large $N$.

In Appendix A.1 we outline an alternative derivation to Eq. (20)—from which Eq. (19) can be inferred—that is specific to the SSR model, and provides additional justification for its large $N$ asymptotic validity. This alternative derivation allows us to find individual expressions for both the output entropy and conditional output entropy. This derivation makes use of the auxiliary PDF, $f_{Q}(\tau)$, derived in Sec. I.3. The significance of this approach is that it leads to our demonstration of the new results that the output entropy can be written for large $N$ as

[TABLE]

while the mutual information can be written as

[TABLE]

where $f_{S}(\cdot)$ is a PDF known as Jeffrey’s prior,

[TABLE]

It is proven in Appendix A.2 that for the SSR model Eq. (23) is indeed a PDF. This is a remarkable result, as in general Jeffrey’s prior has no such simple form. Substitution of Eq. (23) into Eq. (22) and simplifying leads to Eq. (20), which verifies this result.

By inspection of Eq. (19), $f_{S}(x)$ can be derived from knowledge of the noise PDF, $f_{\eta}(\eta)$, since

[TABLE]

II.1 A Sufficient Condition for Optimality

Since relative entropy is always non-negative, from Eq. (22) a sufficient condition for achieving the large $N$ channel capacity is that

[TABLE]

with the resultant capacity as

[TABLE]

Eq. (26) holds provided the conditions for the approximation given by Eq. (20) hold. Otherwise, the RHSs of Eqs. (21) and (22) give lower bounds. This means that for the situations considered previously in Hoch et al. (2003); Stocks (2001b) where the signal and noise both have the same distribution (but different variances), we can expect to find channel capacity that is less than or equal to that of Eq. (26). This is discussed in Sec. III.

The derived sufficient condition of Eq. (25) leads to two ways in which capacity can be achieved, (i) an optimal signal PDF for a given noise PDF, and (ii) an optimal noise PDF for a given signal PDF.

II.2 Optimizing the Signal Distribution

Assuming Eq. (20) holds, the channel capacity achieving input PDF, $f_{X}^{o}(x)$, can be found for any given noise PDF from Eqs. (24) and (25) as

[TABLE]

II.2.1 Example: Uniform Noise

Suppose the iid noise at the input to each threshold device in the SSR model is uniformly distributed on the interval $[-\sigma_{\eta}/2,\sigma_{\eta}/2]$ so that it has PDF

[TABLE]

Substituting Eq. (28) and its associated CDF into Eq. (27), we find that the optimal signal PDF is

[TABLE]

This PDF is in fact the PDF of a sine-wave with uniformly random phase, amplitude $\sigma_{\eta}/2$, and mean $\theta$. A change of variable to the interval $\tau\in[0,1]$ via the substitution $\tau=(x-\theta)/\sigma_{\eta}+0.5$ results in the PDF of the Beta distribution with parameters $0.5$ and $0.5$, also known as the arcsine distribution. As mentioned in Sec. I, this result provides some theoretical justification for the analog-to-digital conversion method proposed in Nguyen (2007).

This Beta distribution is bimodal, with the most probable values of the signal those near zero and unity. Similar results for an optimal input distribution in an information theoretic optimization of a neural system have been found in Schreiber et al. (2002). These results were achieved numerically using the Blahut-Arimoto algorithm often used in information theory to find channel capacity achieving source distributions, or rate-distortion functions Cover and Thomas (1991).

II.2.2 Gaussian Noise

Suppose the iid noise at the input to each threshold device has a zero mean Gaussian distribution with variance $\sigma_{\eta}^{2}$, with PDF

[TABLE]

Substituting Eq. (30) and its associated CDF into Eq. (27), gives the optimal signal PDF. The resultant expression for $f_{X}^{o}(x)$ does not simplify much, and contains the standard error function, $\mbox{erf}(\cdot)$ Spiegel and Liu (1999).

We are able to verify that the resultant PDF has the correct shape via Fig. 8 in Hoch et al. (2003), which presents the result of numerically optimizing the signal PDF, $f_{X}(x)$, for unity variance zero mean Gaussian noise, $\theta=0$, and $N=10000$. As with the work in Schreiber et al. (2002), the numerical optimization is achieved using the Blahut-Arimoto algorithm. It is remarked in Hoch et al. (2003) that the optimal $f_{X}(x)$ is close to being Gaussian. This is illustrated by plotting both $f_{X}(x)$ and a Gaussian PDF with nearly the same peak value as $f_{X}(x)$. It is straightforward to show that a Gaussian with the same peak value as our analytical $f_{X}^{o}(x)$ has variance $0.25\pi^{2}$. If the signal was indeed Gaussian, then we would have $\sigma=2/\pi\simeq 0.6366$, which is very close to the value calculated for actual Gaussian signal and noise in Sec. III.

Our analytical $f_{X}^{o}(x)$ from Eqs. (30) and (27), with $\theta=0$, is plotted on the interval $x\in[-3,3]$ in Fig. 2, along with a Gaussian PDF with variance $0.25\pi^{2}$. Clearly the optimal signal PDF is very close to the Gaussian PDF. Our Fig. 2 is virtually identical to Fig. 8 in Hoch et al. (2003). It is emphasized that the results in Hoch et al. (2003) were obtained using an entirely different method that involves numerical iterations, and therefore provides excellent validation of our theoretical results.

II.3 Optimizing the Noise Distribution

We now assume that the signal distribution is known and fixed. We wish to achieve channel capacity by finding the optimal noise distribution. It is easy to show by integrating Eq. (24) that the CDF corresponding to the PDF, $f_{S}(\cdot)$, evaluated at $x$, can be written in terms of the CDF of the noise distribution as

[TABLE]

If we now let $f_{X}(x)=f_{S}(x)$, then $F_{X}(s)=F_{S}(x)$, and rearranging Eq. (31) gives the optimal noise CDF in terms of the signal CDF as

[TABLE]

Differentiating $F_{\eta}^{o}(x)$ gives the optimal noise PDF as a function of the signal PDF and CDF,

[TABLE]

Unlike optimizing the signal distribution, which is the standard way for achieving channel capacity in information theory Cover and Thomas (1991), we have assumed a signal distribution, and found the ‘best’ noise distribution, which is equivalent to optimizing the channel, rather than the signal.

II.3.1 Example: Uniform Signal

Suppose the signal is uniformly distributed on the interval $x\in[-\sigma_{x}/2,\sigma_{x}/2]$. From Eqs. (32) and (33), the capacity achieving noise distribution has CDF

[TABLE]

and PDF

[TABLE]

Substitution of $F_{\eta}^{o}(x)$ and $f_{\eta}^{o}(x)$ into Eq. (19) finds the interesting result that the Fisher information is constant for all $x$,

[TABLE]

This is verified in Eq. (37) below.

II.4 Consequences of Optimizing the Large $N$ Channel

Capacity

II.4.1 Optimal Fisher Information

Regardless of whether we optimize the signal for given noise, or optimize the noise for a given signal, it is straightforward to show that the Fisher information can be written as a function of the signal PDF,

[TABLE]

Therefore, the Fisher information at large $N$ channel capacity is constant for the support of the signal iff the signal is uniformly distributed. The optimality of constant Fisher information in a neural coding context is studied in Bethge et al. (2002).

II.4.2 The Optimal PDF $f_{Q}(\tau)$

A further consequence that holds in both cases is that the ratio of the signal PDF to the noise PDF is

[TABLE]

This is not a PDF. However, if we make a change of variable via $\tau=1-F_{\eta}(\theta-x)$ we get the PDF $f_{Q}(\tau)$ discussed in Sec. I.3, which for channel capacity is

[TABLE]

This optimal $f_{Q}(\tau)$ is in fact the PDF of the beta distribution with parameters $0.5$ and $0.5$, i.e. the arcsine distribution. It is emphasised that this result holds regardless of whether the signal PDF is optimised for a given noise PDF or vice versa.

II.4.3 Output Entropy at Channel Capacity

From Eq. (I.3.3), the entropy of $Q$ is equal to the negative of the relative entropy between $f_{X}(x)$ and $f_{\eta}(\theta-x)$. The entropy of $Q$ when capacity is achieved can be calculated from Eq. (39) using direct integration as

[TABLE]

From Eqs. (21) and (I.3.3), the large $N$ output entropy at channel capacity in the SSR model is

[TABLE]

II.4.4 The Optimal Output PMF is Beta-Binomial

Suppose we have signal and noise such that $f_{Q}(\tau)=f_{Q}^{o}(\tau)$—i.e. the signal and noise satisfy the sufficient condition, Eq. (25)—but that $N$ is not necessarily large. We can derive the output PMF for this situation, by substituting Eq. (39) into Eq. (17) to get

[TABLE]

where $\beta(a,b)$ is a Beta function. This PMF can be recognized as that of the Beta-binomial—or negative hypergeometric—distribution with parameters $N,0.5$, $0.5$ Evans et al. (2000). It is emphasized that Eq. (II.4.4) holds as an exact analytical result for any $N$.

II.4.5 Analytical Expression for the Mutual Information

The exact expression for the output PMF of Eq. (II.4.4) allows exact calculation of both the output entropy, and the mutual information without need for numerical integration, using Eq. (I.3.2). This is because when $f_{Q}(\tau)=f_{Q}^{o}(\tau)$, the integrals in Eq. (I.3.2) can be evaluated exactly to get

[TABLE]

The exact values of $I_{o}(X,y)$ and the corresponding output entropy, $H_{o}(y)$, are plotted in Fig. 3(a) for $N=1,..,1000$. For comparison, the exact $I(X,y)$ of Eq. (12), which holds for $f_{X}(x)=f_{\eta}(\theta-x)$, is also plotted, as well as the corresponding entropy, $H(y)=\log_{2}{(N+1)}$. It is clear that $I_{o}(X,y)$ is always larger than the mutual information of the $f_{X}(x)=f_{\eta}(\theta-x)$ case, and that $H_{o}(y)$ is always less than its entropy, which is the maximum output entropy.

To illustrate that the large $N$ expressions derived are lower bounds to the exact formula plotted in Fig. 3(a), and that the error between them decreases with $N$, Fig. 3(b) shows the difference between the exact and the large $N$ mutual information and output entropy. This difference clearly decreases with increasing $N$.

II.5 A Note on the Output Entropy

The SSR model has been described in terms of signal quantization theory in McDonnell et al. (2005), and compared with the related process of companding in Amblard et al. (2006). In this context quantization means the conversion of a continuously valued signal to a discretely valued signal that has only a finite number of possible values. Quantization in this sense occurs in analog-to-digital converter circuits, lossy compression algorithms, and in histogram formation Gersho and Gray (1992). For a deterministic scalar quantizer with $N+1$ output states, $N$ threshold values are required. In quantization theory, there is a concept of high resolution quantizers, in which the distribution of $N\rightarrow\infty$ threshold values can be described by a point density function, $\lambda(x)$. For such quantizers, it can be shown that the quantizer output, $y$, in response to a random variable, $X$, has entropy $H(y)\simeq\log_{2}{N}-D(f_{X}||\lambda)$ Gersho and Gray (1992). This is strikingly similar to our Eq. (21) for the large $N$ output entropy of the SSR model. In fact, since the noise that perturbs the fixed threshold value, $\theta$, is additive, each threshold acts as an iid random variable with PDF $f_{\eta}(\theta-x)$, and therefore for large $N$, $f_{\eta}(\theta-x)$ acts as a density function describing the relative frequency of threshold values as a function of $x$, just as $\lambda(x)$ does for a high resolution deterministic quantizer.

For deterministic quantizers, the point density function can be used to approximate the high resolution distortion incurred by the quantization process. For the SSR model however, since the quantization has a random aspect, the distortion has a component due to randomness as well as lossy compression, and cannot be simply calculated from $f_{\eta}(\cdot)$. Instead, one can use the Fisher information to calculate the asymptotic mean square error distortion, which is not possible for deterministic high resolution quantizers.

III Channel Capacity for Large $N$ and ‘Matched’ Signal and

Noise

Unlike the previous section, we now consider channel capacity under the constraint of ‘matched’ signal and noise distributions—i.e. where both the signal and noise, while still independent, have the same distribution, other than their variances. The mean of both signal and noise is zero and the threshold value is also $\theta=0$. In this situation the mutual information depends solely on the ratio $\sigma=\sigma_{\eta}/\sigma_{x}$, which is the only free variable. Finding channel capacity is therefore equivalent to finding the optimal value of noise intensity, $\sigma$. Such an analysis provides verification of the more general capacity expression of Eq. (26), which cannot be exceeded.

Furthermore, inspection of Eq. (65) shows that the large $N$ approximation to the mutual information consists of a term that depends on $N$ and a term that depends only on $\sigma$. This shows that for large $N$ the channel capacity occurs for the same value of $\sigma$—which we denote as $\sigma_{o}$—for all $N$.

This fact is recognized in both Stocks (2001b) for uniform signal and noise—where $\sigma_{o}\rightarrow 1$—and Hoch et al. (2003), for Gaussian signal and noise. Here, we investigate the value of $\sigma_{o}$ and the mutual information at $\sigma_{o}$ for other signal and noise distributions, and compare the channel capacity obtained with the case where $f_{X}(x)=f_{S}(x)$. This comparison finds that the results of Hoch et al. (2003) overstates the true capacity, and that large $N$ results in Stocks (2001b, a) need to be improved to be consistent with the central results of this paper.

From Eq. (22), channel capacity for large $N$ occurs for the value of $\sigma$ that minimizes the relative entropy between $f_{X}$ and $f_{S}$. If we let

[TABLE]

then from Eq. (20), it is also clear that this minimization is equivalent to solving the following problem,

[TABLE]

This is exactly the formulation stated in Hoch et al. (2003). Problem (45) can be equivalently expressed as

[TABLE]

where we have assumed that both the signal and noise PDFs are even functions. The function $f(\sigma)$ can be found for any specified signal and noise distribution by numerical integration, and Problem (46) easily solved numerically. If an exact expression for the relative entropy term is known, then only $g(\sigma)=\int_{x=-\infty}^{x=\infty}f_{X}(x)\log_{2}{(P_{1|x})}dx$ needs to be numerically calculated.

Table 2 gives the result of numerically calculating the value of $\sigma_{o}$, and the corresponding large $N$ channel capacity, $C(X,y)$, for a number of distributions. In each case, $C(X,y)-0.5\log_{2}{(N)}<-0.3956$, as required by Eq. (26). The difference between capacity and $0.5\log_{2}{(N)}$ is about $0.4$ bits per sample. In the limit of large $N$, this shows that capacity is almost identical, regardless of the distribution. However, the value of $\sigma_{o}$ at which this capacity occurs is different in each case.

As discussed in Sec. I.2, the mutual information is identical whenever the signal and noise PDFs are identical, i.e. $\sigma=1$. It is shown below in Eq. (48) that for large $N$ the mutual information at $\sigma=1$ is $I(X,y)=0.5\log_{2}{(N)}-0.6444$. Given that the channel capacity is slightly larger than this, as indicated by Table 2, for each case there is a constant difference between the channel capacity and the mutual information at $\sigma=1$. This value is also listed in Table 2.

III.1 Improvements to Previous Large $N$ Approximations

We now use the results of Sec. II to show that previous large $N$ expressions for the mutual information in the literature for the $\sigma=1$, Gaussian and uniform cases can be improved.

III.1.1 SSR for Large $N$ and $\sigma=1$

We now consider the situation where $f_{X}(x)=f_{\eta}(x)$, so that $\sigma=1$. It is shown in Stocks (2001a) that in this case as $N$ approaches infinity, Eq. (12) reduces to

[TABLE]

To show that this expression can be improved, we begin with the version of Eq. (20) given by Eq. (65). When $\sigma=1$ we have $f_{Q}(\tau)=1$ and $H(Q)=0$. The integrals in Eq. (65) can be solved to give the large $N$ mutual information at $\sigma=1$ as

[TABLE]

Although Eqs. (47) and (48) agree as $N\rightarrow\infty$, the constant terms do not agree. It is shown in Appendix A.3 that the discrepancy can be resolved by improving on the approximation to the average conditional entropy, $H(y|X)$, made in Stocks (2001a). The output entropy at $\sigma=1$ can be shown to be simply $H(y)=\log_{2}{(N+1)}$ Stocks (2001a). Subtracting Eq. (A.3) from $H(y)$ and letting $N$ approach infinity gives

[TABLE]

which does have a constant term which agrees with Eq. (48). The explanation of the discrepancy is that Stocks (2001a) uses the Euler-Maclaurin summation formula to implicitly calculate $\log_{2}{(N!)}$ in the large $N$ approximation to $H(y|X)$. Using Stirling’s approximation for $N!$, as done here, gives a more accurate approximation.

The increased accuracy of Eq. (48) can be verified by numerically comparing both Eq. (48) and Eq. (47) with the exact expression for $I(X,y)$ of Eq. (12), as $N$ increases. The error between the exact expression and Eq. (48) approaches zero as $N$ increases, whereas the error between Eq. (12) and Eq. (47) approaches a nonzero constant for large $N$ of $0.5\log_{2}{\left(\frac{e^{2}}{2\pi}\right)}\simeq 0.117$ bits per sample.

III.1.2 Uniform Signal and

Noise

A derivation is given in Stocks (2001b) of an exact expression for $I(X,y)$ for uniform signal and noise and $\sigma\leq 1$. In addition, Stocks (2001b) finds a large $N$ approximation to the mutual information. Using the same arguments as for the $\sigma=1$ case, this approximation can be improved to

[TABLE]

The accuracy of Eq. (50) can be verified by numerical comparison with the exact formula in Stocks (2001b), as $N$ increases. If one replicates Fig. 3 of Stocks (2001b) in this manner, it is clear that Eq. (50) is the more accurate approximation.

Differentiating Eq. (50) with respect to $\sigma$ and setting to zero obtains the optimal value of $\sigma$ as

[TABLE]

The channel capacity at $\sigma_{o}$ is

[TABLE]

Clearly, $\lim_{N\rightarrow\infty}\sigma_{o}=1$, and the capacity approaches $0.5\log_{2}{\left((N+2)e/(2\pi)\right)}$, which agrees with Eq. (49). Expressions for $\sigma_{o}$ and the corresponding capacity for large $N$ are also given in Stocks (2001b). Again, these are slightly different to Eqs. (51) and (52), due to the slightly inaccurate terms in the large $N$ approximation to $H(y|X)$. However the important qualitative result remains the same, which is that the channel capacity scales with $0.5\log_{2}{(N)}$ and the value of $\sigma$ which achieves this asymptotically approaches unity.

III.1.3 Gaussian Signal and Noise

In Hoch et al. (2003), an analytical approximation for $\sigma_{o}$ for the specific case of Gaussian signal and noise is derived using a Taylor expansion of the Fisher information inside the integral in Eq. (20). We give a slightly different derivation of this approach that uses the PDF $f_{Q}(\tau)$.

We begin with Problem (46). Solving this problem requires differentiating $f(\sigma)$ with respect to $\sigma$ and solving for zero. From Table 1, the derivative of the relative entropy between $f_{X}$ and $f_{\eta}$ is

[TABLE]

For the second term, $g(\sigma)$, we take the lead from Hoch et al. (2003) and approximate $\ln{(P_{1|x})}$ by its second order Taylor series expansion Spiegel and Liu (1999). The result is that

[TABLE]

Numerical testing finds that the approximation of Eq. (54) appears to be quite accurate for all $\sigma$, as the relative error is no more than about $10$ percent for $\sigma>0.2$. However, as we will see, this is inaccurate enough to cause the end result for the approximate channel capacity to significantly overstate the true channel capacity.

Taking the derivative of Eq. (54) with respect to $\sigma$, subtracting it from Eq. (53), setting the result to zero and solving for $\sigma$ gives the optimal value of $\sigma$ found in Hoch et al. (2003), $\sigma_{o}\simeq\sqrt{1-\frac{2}{\pi}}\simeq 0.6028$.

An expression for the mutual information at $\sigma_{o}$ can be found by back-substitution. Carrying this out gives the large $N$ channel capacity for Gaussian signal and noise as

[TABLE]

which can be written as $C(X,y)\simeq 0.5\log_{2}{N}-0.3169$.

Although Eq. (55) is close to correct, recall from Sec. II that capacity must be less than $0.5\log_{2}{N}-0.3956$ and hence Eq. (55) significantly overstates the true capacity.

Acknowledgements.

This work was funded by the Australian Research Council, an Australian Academy of Science Young Researcher’s Award, as well as EPSRC grant EP/C523334/1, and we gratefully acknowledge this support. The authors would also like to thank Priscilla Greenwood, of Arizona State University, Pierre-Olivier Amblard of Laboratoire des Images et des Signaux, France, Thinh Nguyen of Oregon State University and Simon Durrant of the University of Plymouth, UK, for valuable discussions and generous provision of preprints, and the anonymous referees, whose comments have led to significant improvements in this paper.

Appendix A Derivations

A.1 Mutual Information for Large $N$ and Arbitrary

$\sigma$

This appendix contains derivations of the large $N$ approximations to the output entropy and mutual information discussed in Sec. II.

A.1.1 Conditional Output Entropy

An approximation to the conditional output entropy, $H(y|X)$, can be derived by noting that for large $N$ the binomial distribution can be approximated by a Gaussian distribution with the same mean and variance—i.e. $NP_{1|x}$ and $NP_{1|x}(1-P_{1|x})$ respectively. Provided $0{\ll}NP_{1|x}{\ll}N$ we have

[TABLE]

The average conditional output entropy is $H(y|X)=\int_{x}f_{X}(x)\hat{H}(y|x)dx$, where

[TABLE]

Using the well known result for the entropy of a Gaussian random variable Cover and Thomas (1991) we can write

[TABLE]

Multiplying both sides of Eq. (58) by $f_{X}(x)$ and integrating over all $x$ gives

[TABLE]

Eq. (A.1.1) can be verified for the case where $f_{X}(x)=f_{\eta}(\theta-x)$, since this means $f_{Q}(\tau)=1$ and $\int_{\tau=0}^{\tau=1}f_{Q}(\tau)\log_{2}{(\tau)}d\tau=-\log_{2}{(e)}$. Consequently Eq. (A.1.1) reduces to $H(y|X)\simeq 0.5\log_{2}{\left(\frac{2{\pi}N}{e}\right)}$ which agrees precisely with Eq. (74). This approximation breaks down when $P_{1|x}$ is close to zero or unity. Furthermore, Eq. (A.1.1) holds exactly only for values of $x$ for which $P_{y|X}(n|x)$ is exactly Gaussian. Otherwise, $H(y|X)$ is strictly less than the approximation given.

A.1.2 Output Distribution and entropy

For large $N$, since $P_{y|X}(n|x)$ is Gaussian, $y/N$ approaches a delta function located at $P_{1|x}=n/N$. From Eqs. (7) and (17), this means that $P_{y}(n)$ can be written in terms of the PDF of the average transfer function, $f_{Q}(\cdot)$, as

[TABLE]

This result can be derived more rigorously using saddlepoint methods Brunel and Nadal (1998).

Consider the case where the signal and noise both have the same distribution but different variances. When the noise intensity, $\sigma>1$, then $f_{Q}(0)=f_{Q}(1)=0$, whereas for $\sigma<1$, we have $f_{Q}(0)=f_{Q}(1)=\infty$. From Eq. (60), this means $P_{y}(0)$ and $P_{y}(N)$ are either zero or infinite. However, for finite $N$, there is some finite nonzero probability that all output states are on or off. Indeed, at $\sigma=1$, we know that $P_{y}(n)=\frac{1}{N+1}~{}\forall~{}n$, and at $\sigma=0$, $P_{y}(0)=P_{y}(N)=0.5$. Furthermore, for finite $N$, Eq. (60) does not guarantee that $\sum_{n=0}^{N}P_{y}(n)=1$. To increase the accuracy of our approximation by ensuring $P_{y}(0)$ and $P_{y}(N)$ are always finite, and that $P_{y}(n)$ forms a valid PMF, we define a new approximation as

[TABLE]

Fig. 4 shows that the approximation given by $P_{y}^{\prime}(n)$ is highly accurate for $N$ as small as $63$, for $\sigma$ both smaller and larger than unity.

Consider the entropy of the discrete random variable $y$. Making use of Eq. (60), we have

[TABLE]

Suppose that the summations above can be approximated by integrals, without any remainder terms. Carrying this out and then making the change of variable $\tau=n/N$ gives

[TABLE]

where $H(Q)$ is the differential entropy of the random variable $Q$. Performing a change of variable in Eq. (A.1.2) of $\tau=1-F_{\eta}(\theta-x)$ gives

[TABLE]

This result shows that $H(y)$ for large $N$ is approximately the sum of the number of output bits and the negative of the relative entropy between $f_{X}$ and $f_{\eta}$. Therefore, since relative entropy is always non-negative, the approximation to $H(y)$ given by Eq. (64) is always less than or equal to $\log_{2}{(N)}$. This agrees with the known expression for $H(y)$ in the specific case of $\sigma=1$ of $\log_{2}{(N+1})$, which holds for any $N$.

A.1.3 Mutual Information

Subtracting Eq. (A.1.1) from Eq. (A.1.2) gives a large $N$ approximation to the mutual information as

[TABLE]

As discussed in the main text, the mutual information scales with $0.5\log_{2}{(N)}$. The importance of the $N$-independent terms in Eq. (65) is that they determine how the mutual information varies from $0.5\log_{2}{\left(\frac{N}{2{\pi}e}\right)}$ for different PDFs, $f_{Q}(\tau)$.

Fig. 5 shows, as examples, the approximation of Eq. (65), as well as the exact mutual information—calculated by numerical integration—for the Gaussian and Laplacian cases, for a range of $\sigma$ and increasing $N$. As with the output entropy, the mutual information approximation is quite good for $\sigma>0.7$, but worsens for smaller $\sigma$. However, as $N$ increases the approximation improves.

Eq. (65) can be rewritten via the change of variable, $x=\theta-F_{\eta}^{-1}(1-\tau)$, as

[TABLE]

Rearranging Eq. (A.1.3) gives Eq. (20)—with the Fisher information, $J(x)$, given by Eq. (19)—which is precisely the same as that derived in Hoch et al. (2003) as an asymptotic large $N$ expression for the mutual information. Our analysis extends Hoch et al. (2003) by finding large $N$ approximations to both $H(y)$ and $H(y|X)$, as well as the output distribution, $P_{y}(n)$. We have also illustrated the role of the PDF, $f_{Q}(\tau)$, in these approximations, and justified the use of Eq. (20) for the SSR model.

A.2 Proof that $f_{S}(x)$ is a PDF

As shown in Hoch et al. (2003), the Fisher information for the SSR model is given by Eq. (19). Consider $f_{S}(x)$ as in Eq. (24). Since $f_{\eta}(x)$ is a PDF and $F_{\eta}(x)$ is the CDF of $\eta$ evaluated at $x$, we have $f_{S}(x)\geq 0~{}\forall~{}x$. Letting $h(x)=F_{\eta}(\theta-x)$, Eq. (24) can be written as

[TABLE]

Suppose $f_{\eta}(x)$ has support $x\in[-a,a]$. Integrating $f_{S}(x)$ over all $x$ gives

[TABLE]

which means $f_{S}(x)$ is a PDF.

A.3 $H(y|X)$ for large $N$ and

$\sigma=1$

Here we derive a large $N$ approximation to $H(y|X)$ used in Sec. III.1.1. For $\sigma=1$ the output PMF is $P_{y}(n)=\frac{1}{N+1}$ $\forall$ $n$ Stocks (2001a). Using this, it can be shown that

[TABLE]

We will now see that both terms of Eq. (69) can be simplified by approximations that hold for large $N$. Firstly, for the $\log_{2}{(N!)}$ term, we can make use of Stirling’s formula Spiegel and Liu (1999), which is valid for large $N$,

[TABLE]

This approximation is particularly accurate if the log is taken of both sides, which we require. Secondly, the sum in the second term of Eq. (69) can be approximated by an integral and simplified by way of the Euler-Maclaurin summation formula Spiegel and Liu (1999). The result is

[TABLE]

Subtracting Eq. (71) from the log of Eq. (70) gives

[TABLE]

where we have used $N\log_{2}{(1+\frac{1}{N})}=\frac{1}{\ln{2}}+O\left(\frac{1}{N}\right)$. When Eq. (A.3) is substituted into an exact expression for $H(y|X)$ given in Stocks (2001a), we get

[TABLE]

The final result is that for large $N$,

[TABLE]