The Relation Between Bayesian Fisher Information and Shannon Information   for Detecting a Change in a Parameter

Eric Clarkson

arXiv:1902.00103·cs.IT·July 24, 2019

The Relation Between Bayesian Fisher Information and Shannon Information for Detecting a Change in a Parameter

Eric Clarkson

PDF

TL;DR

This paper establishes a theoretical connection between Shannon Information, Fisher Information, and Bayesian Fisher Information in the context of detecting changes in a parameter, linking estimation and detection performance measures.

Contribution

It introduces a novel theoretical framework linking Shannon Information and Fisher Information through Bayesian Fisher Information for detection tasks.

Findings

01

Derived a relationship between Shannon Information and Fisher Information.

02

Connected Bayesian Fisher Information to detection performance metrics.

03

Outlined a cycle of relations involving Minimum Probability of Error and Mean Squared Error.

Abstract

We derive a connection between performance of estimators the performance of the ideal observer on related detection tasks. Specifically we show how Shannon Information for the task of detecting a change in a parameter is related to the Fisher Information and the Bayesian Fisher Information. We have previously shown that this Shannon Information is related via an integral transform to the Minimum Probability of Error on the same task. We then outline a circle of relations starting with this Minimum Probability of Error and Ensemble Mean Squared Error via the Ziv-Zakai inequality, then the Ensemble Mean Squared error for an estimator and the Bayesian Fisher Information via the van Trees Inequality, and finally the Bayesian Fisher Information and the Shannon Information for a detection task via the work here.

Equations122

F (θ) = ⟨ [\frac{d}{d θ} ln p r (g ∣ θ)]^{2} ⟩_{g ∣ θ} .

F (θ) = ⟨ [\frac{d}{d θ} ln p r (g ∣ θ)]^{2} ⟩_{g ∣ θ} .

v a r (\hat{θ}) \geq [F (θ)]^{- 1}

v a r (\hat{θ}) \geq [F (θ)]^{- 1}

⟨ \hat{θ} (g) ⟩_{g ∣ θ} = θ .

⟨ \hat{θ} (g) ⟩_{g ∣ θ} = θ .

Λ (g) = \frac{p r ( g ∣ θ ~ )}{p r ( g ∣ θ )}

Λ (g) = \frac{p r ( g ∣ θ ~ )}{p r ( g ∣ θ )}

A U C (θ, \tilde{θ}) = \frac{1}{2} + \frac{1}{2} erf [\frac{1}{2} d (θ, \tilde{θ})]

A U C (θ, \tilde{θ}) = \frac{1}{2} + \frac{1}{2} erf [\frac{1}{2} d (θ, \tilde{θ})]

d^{2} (θ, θ + △ θ) = (△ θ)^{2} F (θ) + \dots

d^{2} (θ, θ + △ θ) = (△ θ)^{2} F (θ) + \dots

F = ⟨ F (θ) ⟩_{θ} + ⟨ [\frac{d}{d θ} ln p r (θ)]^{2} ⟩_{θ} .

F = ⟨ F (θ) ⟩_{θ} + ⟨ [\frac{d}{d θ} ln p r (θ)]^{2} ⟩_{θ} .

p r (θ ∣ g) = \frac{p r ( g ∣ θ ) p r ( θ )}{p r ( g )},

p r (θ ∣ g) = \frac{p r ( g ∣ θ ) p r ( θ )}{p r ( g )},

p r (g) = \int_{- \infty}^{\infty} p r (g ∣ θ) p r (θ) d θ .

p r (g) = \int_{- \infty}^{\infty} p r (g ∣ θ) p r (θ) d θ .

F = ⟨ ⟨ [\frac{d}{d θ} ln p r (θ ∣ g)]^{2} ⟩_{g ∣ θ} ⟩_{θ} .

F = ⟨ ⟨ [\frac{d}{d θ} ln p r (θ ∣ g)]^{2} ⟩_{g ∣ θ} ⟩_{θ} .

E M S E = ⟨ ⟨ [\hat{θ} (g) - θ]^{2} ⟩_{g ∣ θ} ⟩_{θ}

E M S E = ⟨ ⟨ [\hat{θ} (g) - θ]^{2} ⟩_{g ∣ θ} ⟩_{θ}

E M S E \geq F^{- 1} .

E M S E \geq F^{- 1} .

P r_{0} (θ, \tilde{θ}) = \frac{p r ( θ )}{p r ( θ ) + p r ( θ ~ )}

P r_{0} (θ, \tilde{θ}) = \frac{p r ( θ )}{p r ( θ ) + p r ( θ ~ )}

P r_{1} (θ, \tilde{θ}) = \frac{p r ( θ ~ )}{p r ( θ ) + p r ( θ ~ )} .

P r_{1} (θ, \tilde{θ}) = \frac{p r ( θ ~ )}{p r ( θ ) + p r ( θ ~ )} .

Λ_{0} = y (θ, \tilde{θ}) = \frac{P r _{0} ( θ , θ ~ )}{P r _{1} ( θ , θ ~ )} = \frac{p r ( θ )}{p r ( θ ~ )} .

Λ_{0} = y (θ, \tilde{θ}) = \frac{P r _{0} ( θ , θ ~ )}{P r _{1} ( θ , θ ~ )} = \frac{p r ( θ )}{p r ( θ ~ )} .

E M S E \geq \frac{1}{2} \int_{0}^{\infty} \int_{- \infty}^{\infty} [p r (θ) + p r (θ + x)] P_{e} (θ, θ + x) d θ x d x

E M S E \geq \frac{1}{2} \int_{0}^{\infty} \int_{- \infty}^{\infty} [p r (θ) + p r (θ + x)] P_{e} (θ, θ + x) d θ x d x

E M S E \geq \frac{1}{2} \int_{- \infty}^{\infty} \int_{θ}^{\infty} [p r (θ) + p r (\tilde{θ})] P_{e} (θ, \tilde{θ}) (\tilde{θ} - θ) d \tilde{θ} d θ .

E M S E \geq \frac{1}{2} \int_{- \infty}^{\infty} \int_{θ}^{\infty} [p r (θ) + p r (\tilde{θ})] P_{e} (θ, \tilde{θ}) (\tilde{θ} - θ) d \tilde{θ} d θ .

E M S E \geq \frac{1}{2} \int_{- \infty}^{\infty} \int_{θ}^{\infty} [p r (θ) + p r (\tilde{θ})] P_{e} (θ, \tilde{θ}) \tilde{θ} - θ d \tilde{θ} d θ

E M S E \geq \frac{1}{2} \int_{- \infty}^{\infty} \int_{θ}^{\infty} [p r (θ) + p r (\tilde{θ})] P_{e} (θ, \tilde{θ}) \tilde{θ} - θ d \tilde{θ} d θ

E M S E \geq \frac{1}{2} \int_{- \infty}^{\infty} \int_{- \infty}^{\tilde{θ}} [p r (θ) + p r (\tilde{θ})] P_{e} (θ, \tilde{θ}) \tilde{θ} - θ d θ d \tilde{θ}

E M S E \geq \frac{1}{2} \int_{- \infty}^{\infty} \int_{- \infty}^{\tilde{θ}} [p r (θ) + p r (\tilde{θ})] P_{e} (θ, \tilde{θ}) \tilde{θ} - θ d θ d \tilde{θ}

E M S E \geq \frac{1}{2} \int_{- \infty}^{\infty} \int_{- \infty}^{θ} [p r (θ) + p r (\tilde{θ})] P_{e} (θ, \tilde{θ}) \tilde{θ} - θ d \tilde{θ} d θ .

E M S E \geq \frac{1}{2} \int_{- \infty}^{\infty} \int_{- \infty}^{θ} [p r (θ) + p r (\tilde{θ})] P_{e} (θ, \tilde{θ}) \tilde{θ} - θ d \tilde{θ} d θ .

E M S E \geq \frac{1}{4} \int_{- \infty}^{\infty} \int_{- \infty}^{\infty} [p r (θ) + p r (\tilde{θ})] P_{e} (θ, \tilde{θ}) \tilde{θ} - θ d \tilde{θ} d θ

E M S E \geq \frac{1}{4} \int_{- \infty}^{\infty} \int_{- \infty}^{\infty} [p r (θ) + p r (\tilde{θ})] P_{e} (θ, \tilde{θ}) \tilde{θ} - θ d \tilde{θ} d θ

E M S E \geq \frac{1}{2} \int_{- \infty}^{\infty} \int_{- \infty}^{\infty} p r (θ) P_{e} (θ, \tilde{θ}) \tilde{θ} - θ d \tilde{θ} d θ .

E M S E \geq \frac{1}{2} \int_{- \infty}^{\infty} \int_{- \infty}^{\infty} p r (θ) P_{e} (θ, \tilde{θ}) \tilde{θ} - θ d \tilde{θ} d θ .

E M S E \geq \frac{1}{2} ⟨ \int_{- \infty}^{\infty} P_{e} (θ, \tilde{θ}) \tilde{θ} - θ d \tilde{θ} ⟩_{θ} .

E M S E \geq \frac{1}{2} ⟨ \int_{- \infty}^{\infty} P_{e} (θ, \tilde{θ}) \tilde{θ} - θ d \tilde{θ} ⟩_{θ} .

Λ (g ∣ θ, \tilde{θ}) = \frac{p r ( g ∣ θ ~ )}{p r ( g ∣ θ )}

Λ (g ∣ θ, \tilde{θ}) = \frac{p r ( g ∣ θ ~ )}{p r ( g ∣ θ )}

I (y) = \frac{1}{1 + y} ⟨ ln [\frac{Λ ( 1 + y )}{Λ + y}] ⟩_{g ∣ \tilde{θ}} + \frac{y}{1 + y} ⟨ ln [\frac{1 + y}{Λ + y}] ⟩_{g ∣ θ} .

I (y) = \frac{1}{1 + y} ⟨ ln [\frac{Λ ( 1 + y )}{Λ + y}] ⟩_{g ∣ \tilde{θ}} + \frac{y}{1 + y} ⟨ ln [\frac{1 + y}{Λ + y}] ⟩_{g ∣ θ} .

H (y) = - \frac{1}{1 + y} ln (\frac{1}{1 + y}) - \frac{y}{1 + y} ln (\frac{y}{1 + y}) .

H (y) = - \frac{1}{1 + y} ln (\frac{1}{1 + y}) - \frac{y}{1 + y} ln (\frac{y}{1 + y}) .

C_{e} (y) = \frac{1}{1 + y} ⟨ ln (\frac{Λ + y}{Λ}) ⟩_{g ∣ \tilde{θ}} + \frac{y}{1 + y} ⟨ ln (\frac{Λ + y}{y}) ⟩_{g ∣ θ} .

C_{e} (y) = \frac{1}{1 + y} ⟨ ln (\frac{Λ + y}{Λ}) ⟩_{g ∣ \tilde{θ}} + \frac{y}{1 + y} ⟨ ln (\frac{Λ + y}{y}) ⟩_{g ∣ θ} .

C_{e} (θ, \tilde{θ}) = C_{e} (y) = \frac{1}{1 + y} [⟨ (Λ + y) ln (Λ + y) ⟩_{g ∣ θ} - ⟨ Λ ln Λ ⟩_{g ∣ θ} - y ln y] .

C_{e} (θ, \tilde{θ}) = C_{e} (y) = \frac{1}{1 + y} [⟨ (Λ + y) ln (Λ + y) ⟩_{g ∣ θ} - ⟨ Λ ln Λ ⟩_{g ∣ θ} - y ln y] .

C_{e}^{'} (θ) = \frac{d}{d θ ~} C_{e} (θ, \tilde{θ})_{\tilde{θ} = θ} .

C_{e}^{'} (θ) = \frac{d}{d θ ~} C_{e} (θ, \tilde{θ})_{\tilde{θ} = θ} .

Λ^{'} = Λ^{'} (g ∣ θ) = \frac{d}{d θ ~} Λ (g ∣ θ, \tilde{θ})_{\tilde{θ} = θ} = \frac{p r ^{'} ( g ∣ θ )}{p r ( g ∣ θ )},

Λ^{'} = Λ^{'} (g ∣ θ) = \frac{d}{d θ ~} Λ (g ∣ θ, \tilde{θ})_{\tilde{θ} = θ} = \frac{p r ^{'} ( g ∣ θ )}{p r ( g ∣ θ )},

Peer Reviews

No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.

Videos

No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.

Full text

The Relation Between Bayesian Fisher Information and Shannon Information

for Detecting a Change in a Parameter

Eric Clarkson

Abstract

We derive a connection between performance of estimators the performance of the ideal observer on related detection tasks. Specifically we show how Shannon Information for the task of detecting a change in a parameter is related to the Bayesian Fisher Information. We have previously shown that this Shannon Information is related via an integral transform to the Minimum Probability of Error on the same task. We then outline circle of relations starting with this Minimum Probability of Error and Ensemble Mean Squared Error via the Ziv-Zakai inequality, then the Ensemble Mean Squared error for an estimator and the Bayesian Fisher Information via the van Trees Inequality, and finally the Bayesian Fisher Information and the Shannon Information for a detection task via the work here.

1 Introduction

Fisher Information (FI) is normally thought of as related to the performance of an estimator of a parameter in a statistical model for the given data. The ordinary FI is related through the Cramer-Rao Bound (CRB) to the variance of an unbiased estimator of the parameter. what we will call the Bayesian FI (BFI) is related by the van Tress inequality to the Ensemble Mean Squared Error (EMSE) for an estimator. On the other hand we have shown previously that the FI is related to the performance of the ideal observer on the task of detecting a small change in the parameter, as measured by the area under the Receiver Operating Characteristic (ROC) curve. Thus FI is related to performance on a binary classification task. The Ziv-Zakai inequality relates the EMSE of an estimator to the performance of the ideal observer on the task of distinguishing two different values of the parameter as measured by the minimum probability of error (MPE). Thus EMSE is also related to performance on a binary classification task. In this work we will derive further connections between performance of estimators the performance of the ideal observer on related detection tasks. Specifically we will show how Shannon Information (SI) for a certain binary classification task is related to the BFI. We have previously shown that this SI is related via an integral transform to the MPE on the same task. We then have a circle of relations starting with MPE and EMSE via Ziv-Zakai, then EMSE and BFI via van Trees, BFI and SI via the work here, and finally SI and MPE via the aforementioned integral transform.

In Section 2 we have a brief review of FI and the CRB. Section 3 contains a previously derived result relating the area ubder the ideal observer ROC curve to FI when the task is to detect a small change in the parameter. The reason for showing this result here is to emphasize the similarity with the results below relating SI with BFI. In Section 4 we define the BFI and briefly discuss the van Trees inequality. The Ziv-Zakai inequality is presented in Section 5, and we have a short calculation that derives a more compact form that is easily related to other results here. Section 6 presents a derivation of the main result which relates the conditional entropy for the task of detecting a small change in a parameter to the BFI. In Section 7 we show the vector-parameter version of the main result. The main result is reformulatd in terms of SI in Section 8 and our conclusions are stated in Section 9.

2 Fisher information

We will consider a scalar parameter $\theta$ throughout. This parameter takes values on the real line. There are also vector parameter versions of everything. The data is a random vector $\mathbf{g}$ generated by a noisy imaging system, and has a conditional probability density function (PDF) denoted by $pr\left(\mathbf{g}|\theta\right)$ . The Fisher Information (FI) is then given by [1,2]

[TABLE]

Suppose that $\hat{\theta}\left(\mathbf{g}\right)$ is an estimator that uses the data vector to produce an estimate of $\theta$ . The Cramer-Rao Bound (CRB) states that

[TABLE]

when

[TABLE]

In other words the variance of an unbiased estimator is bounded below by the reciprocal of the FI. For the Bayesian FI there is the van Trees inequality which is similar to the Cramer-Rao lower bound. This will be discussed further below.

3 FI and AUC

We consider a classification task to classify $\mathbf{g}$ as a sample drawn from from $pr\left(\mathbf{g}|\theta\right)$ or $pr\left(\mathbf{g}|\tilde{\theta}\right)$ , corresponding to hypotheses $H_{0}$ and $H_{1}$ respectively. The ideal-observer AUC for this task computes the likelihood ratio [3]

[TABLE]

and compare to a threshold $\Lambda_{0}$ in order to make the decision. If the likelihood ratio is greater than the threshold, then the decision $D_{1}$ is made that $\mathbf{g}$ was drawn from from $pr\left(\mathbf{g}|\tilde{\theta}\right)$ . If the likelihood ratio is less than or equal to the threshold, then the decision $D_{0}$ is made that $\mathbf{g}$ was drawn from from $pr\left(\mathbf{g}|\theta\right)$ . The True Positive Fraction (TPF) is $Pr\left(D_{1}|H_{1}\right)$ , the probability that the decision $D_{1}$ is made when $H_{1}$ is true. The False Positive Fraction (FPF) is $Pr\left(D_{1}|H_{0}\right)$ , the probability that the decision $D_{1}$ is made when $H_{0}$ is true. The Receiver Operating Characterisitc (ROC) curve for this task plots TPF versus FPF as the threshold is varied from [math] to $\infty$ . This is a concave curve that starts at the point $\left(1,1\right)$ and ends at $\left(0,0\right)$ . The area under the ideal-observer ROC curve, the AUC, is a commonly used figure of merit (FOM) for an imaging system on a classification task. In this case we will denote this quantity by $AUC\left(\theta,\tilde{\theta}\right)$ . For the ideal observer the AUC is always between $0.5$ and $1$ . This area is related to the detectability $d\left(\theta,\tilde{\theta}\right)$ by the equation

[TABLE]

The detectability is an equivalent FOM to the AUC and varies from [math] to $\infty$ . In a previous publication we showed that that the Taylor series expansion for the ideal-observer detectability in this situation is given by [4,5,6]

[TABLE]

Our goal in this work is to derive a similar relation between the Bayesian FI and the average Shannon Information (SI) for the classification task that we have defined here.

4 Bayesian FI and EMSE

For the rest of this paper we will assume that we have a prior probability $pr\left(\theta\right)$ on the parameter of interest. The Bayesian FI is given by

[TABLE]

For an alternate expression of the Bayesian FI we define the posterior PDF $pr\left(\theta|\mathbf{g}\right)$ on $\theta$ by

[TABLE]

where

[TABLE]

In terms of the posterior PDF we can show that

[TABLE]

The Ensemble Mean Squared Error (EMSE) for any estimator of $\theta$ is defined by

[TABLE]

The van Trees inequality states that [7,8]

[TABLE]

In other words, the EMSE for any estimator of the parameter of interest is bounded below by the reciprocal of the Bayesian FI. The van Tress inequality is also called the Bayesian CRB. This is the reason for calling $F$ the Bayesian FI. This inequality, much like the CRB, relates the relevant version of the FI to performance on an estimation task. On the other hand, the result described in Section 2 relates the FI to a classification task, namely the detection of a small change in the parameter of interest. We want to connect the Bayesian FI to this same task, but before we do this we discuss a relation between the EMSE and the detection task in Section 2 .

5 Ziv-Zakai inequality (EMSE and MPE)

When the prior $pr\left(\theta\right)$ is known, then we can define prior probabilities for the two hypotheses $H_{0}$ and $H_{1}$ for the detection task in Section 2 via

[TABLE]

and

[TABLE]

Then the minimum probabilty of error $P_{e}\left(\theta,\tilde{\theta}\right)$ on the this detection task is the probability of error for the ideal observer when the threshold used for the likelihood ratio is given by

[TABLE]

The minimum probability of error is $P_{e}\left(\theta,\tilde{\theta}\right)=\left(FPF\right)Pr_{0}+\left(1-TPF\right)Pr_{1}$ since $1-TPF$ is the False Negative Fraction (FNF) $Pr\left(D_{0}|H_{1}\right)$ . Of course, in this expression, $TPF$ and $FPF$ also depend on the pair $\left(\theta,\tilde{\theta}\right)$ . The Ziv-Zakai inequality, in its standard form, states that [9,10]

[TABLE]

We will show that this inequality can be written in an alternate form which connects it to the classification task in Section 2. We start with the substitution $\tilde{\theta}=\theta+x$ so that we have

[TABLE]

Due to the range of integration for $\tilde{\theta}$ we may also write this as

[TABLE]

Interchanging the order of integration we have

[TABLE]

Now we note that $P_{e}\left(\theta,\tilde{\theta}\right)=P_{e}\left(\tilde{\theta},\theta\right)$ since the two tasks described by the pairs $\left(\theta,\tilde{\theta}\right)$ and $\left(\tilde{\theta},\theta\right)$ are the same task. Therefore we may interchange the variables $\theta$ and $\tilde{\theta}$ and use the symmetry of all of the functions in the integrand to arrive at

[TABLE]

Now combining the second and fourth inequalities in this chain we have

[TABLE]

We separate the right hand side into two integrals and again use the symmetry in $\theta$ and $\tilde{\theta}$ to find out that the two integrals are the same. Therefore we have the final result

[TABLE]

Finally, the integral on the right can be written as an expectation:

[TABLE]

This formulation of the Ziv-Zakai inequality makes it clear that the worst case scenario, from the EMSE persepctive, is when the minimum proabablity of error $P_{e}\left(\theta,\tilde{\theta}\right)$ on the classification task in Section 2 is significant for relatively large values of $\left|\tilde{\theta}-\theta\right|$ . In the next section we will complete the circle by relating the Bayesian FI to performance on the classification task in Section 2 as measured by the average SI.

6 SI and the Bsyesian FIM

For the purposes of this section we will use the notation

[TABLE]

for the likelihood ratio for the classification task in Section 2. Keeping in mind that $\Lambda=\Lambda\left(\mathbf{g}|\theta,\tilde{\theta}\right)$ and $y=y\left(\theta,\tilde{\theta}\right)$ , we can write the SI for this classification task as [11]

[TABLE]

If we introduce a binary random variable $X$ such that $X=1$ with probability $Pr_{1}\left(\theta,\tilde{\theta}\right)$ , and $X=0$ with probability $Pr_{0}\left(\theta,\tilde{\theta}\right)$ , then the entropy of $X$ is given by

[TABLE]

The conditional entropy for $X$ given $\mathbf{g}$ is then defined by $C_{e}\left(y\right)=H\left(y\right)-I\left(y\right).$ The non-standard notation for conditional entropy here was introduced in an earlier publication that related conditional entropy to MPE via an integral transform in the variable $y$ . After some algebra we have the expression

[TABLE]

We are interested in the limiting value of the conditional entropy as $\tilde{\theta}$ approaches $\theta$ . After some more simplification involving the definition of the likelihood ratio we can write the conditioal entropy for the task in Section 2 in the form

[TABLE]

Note that only expectations under hypothesis $H_{0}$ are needed to compute the conditional entropy, and hence the SI.

In order to keep the notational complications to a minimum we introduce the function

[TABLE]

We also will use the following notation

[TABLE]

where the prime on the far right indicates a derivative with respect to $\theta$ . Similarly we will write

[TABLE]

Now, using the fact that, when $\tilde{\theta}=\theta$ , we have $y=1$ and $\Lambda=1$ ,

[TABLE]

Now we use the fact that $\left\langle\Lambda^{\prime}\right\rangle_{\mathbf{g}|\theta}=0$ to give us

[TABLE]

This result is to be expected since, as a function of $\tilde{\theta}$ , the conditional entropy $C_{e}\left(\theta,\tilde{\theta}\right)$ is maximized when $\tilde{\theta}=\theta$ .

Now we will look at second derivative terms. We define

[TABLE]

We also will use the following notation

[TABLE]

and

[TABLE]

In order to ease the mental computations for the reader we note that, for any variable $z$ that depends on $\theta$ , we have

[TABLE]

and

[TABLE]

Also note that we use

[TABLE]

and

[TABLE]

Now using the Liebniz rule for second derivatives we have the expansion $C_{e}^{\prime\prime}\left(\theta\right)=A+2B+C,$ where the terms on the right are given by

[TABLE]

and

[TABLE]

These three terms simplify algebraically to

[TABLE]

and

[TABLE]

Now we use the fact that $\left\langle\Lambda^{{}^{\prime\prime}}\right\rangle_{g|\theta}=0$ and $\left\langle\Lambda^{\prime}y^{\prime}\right\rangle_{g|\theta}=y^{\prime}\left\langle\Lambda^{\prime}\right\rangle_{g|\theta}=0$ to arrive at

[TABLE]

If we use the prior to average over $\theta$ we then have

[TABLE]

Since $\left\langle C_{e}\left(\theta,\theta\right)\right\rangle_{\theta}=\left\langle\ln 2\right\rangle_{\theta}=\ln 2$ we have the Taylor expansion of the average conditional entropy for the task in Section 2:

[TABLE]

Thus the Bayesian FI gives the lowest order approximation for the average conditional entropy when the task is detecting a small change in the parameter of interest.

The Ziv-Zakai inequality provides a relation between the MPE for the task of using the data vector $\mathbf{g}$ to detect a change in a parameter $\theta$ in the conditional PDF $pr\left(\mathbf{g}|\theta\right)$ , and the EMSE for the task of using $\mathbf{g}$ to estimate a parameter $\theta$ in the conditional PDF $pr\left(\mathbf{g}|\theta\right)$ . The van Trees inequality relates this EMSE the Bayesian FI. We have now found a relationship between the Bayesian FI and the average conditional entropy for the detection task. This completes the circle since there is an integral relation between the conditionaal entropy for any classification task and the MPE for the same task [12,13]. This reinforces the idea that there is a strong mathematical relationship between the performance of an imaging system on estimation tasks and the performance of the same system on the detection of a small change in the parameter of interest.

7 Vector version

For athe vector version of this last result we assume a vector parameter $\boldsymbol{\theta}$ and a perturbation in this parameter of the form $t\mathbf{u}$ , where $\mathbf{u}$ is a unit vector in parameter space. We then have a Taylor series expansion in the varaible $t$ given by

[TABLE]

in this equation the Bayesian Fisher Information Matrix (FIM) is given by

[TABLE]

where the ordinary FIM is defined by

[TABLE]

There are vector versions of the van Trees inequality and the Ziv-Zakai inequality so the circle of relations between EMSE, MPE and Bayesian FIM also exists for vector parameters..

8 Taylor series expansion for SI

To get a Taylor series expansion for SI similar to the one we have derived for conditional entropy we must find the expansion for $H\left(y\right)$ . This function can be written in the form

[TABLE]

Using notation similar to that in Section 5 we have

[TABLE]

For the second derivative we can write $H^{\prime\prime}\left(\theta\right)=a+2b+c$ , with

[TABLE]

and

[TABLE]

Combining terms we have

[TABLE]

Now we may combine these computations with the expansion in Section 5 and find that

[TABLE]

This gives us another relation beween the FI and the detection task in Section 2. The Taylor series expansion for $I\left(\theta,\theta+\triangle\theta\right)$ now has the form

[TABLE]

Thus the SI and the ideal-observer detectability are essentially the same FOM when we are trying to detect a small change in a parameter.

After using the prior on $\theta$ to compute an expectation we have

[TABLE]

Thus the quantity $\left\langle F\left(\theta\right)\right\rangle_{\theta}$ also has an interpretation in terms of the average SI for the detection task in Section 2.

9 Conclusion

Fisher Information (FI) is almost always thought of in terms of the cramer-Rao Bound. What we call the Bayesian Fisher Information is usually thought of in terms of the van Trees inequality. We have shown previously that Fisher Information is related to the performance of the ideal observer on the task of detecting a small change in the parameter, as measured by the area under the Receiver Operating Characteristic curve. Thus Fisher Information is related to the ideal performance on a binary classification task. The Ziv-Zakai inequality relates the Ensemble Mean Squared Error of an estimator to the performance of the ideal observer on the task of distinguishing two different values of the parameter, as measured by the Minimum Probability of Error. In this work we showed how Shannon Information for a certain binary classification task is related to the Bayesian Fisher Information. Since we have previously shown that this Shannon Information is related via an integral transform to the Minimum Probability of Error on the same task, we then have a circle of relations relating Minimum Probability of Error on a detection tsak, Ensemble Mean Squared Error of an Estimator, Bayesian Fisher Information, and Shannon Information for the detection task.

One of the useful results of this work is that Tsak-Based Shannon Information (TSI) for the detection of a small change in a parameter and the ideal-observer AUC for the same task at essentially equivalent figures of merit for a given imaging system. Optimizing asystem for one of these quantities will also optimoze it for the other. Another point of interest is that, when the parameter is a vector, the interpretation of the Bayesian Fisher Information Matrix in terms of Shannon Information does not require inversion of the matrix. This is similar to the connection betwee the Fisher Information Matrix and the ideal-observer AUC derived previously. Finally, both of these relations are the first terms iof a Taylor series expansion, and not inequalities as the Cramer-Rap Bound and the van Trees inequality are. They thus have the potential to provide good approximations to the relevant TSI or AUC values.

Bibliography13

The reference list from the paper itself. Each links out to its DOI / PubMed record.

1[1] J. Shao , Mathematical Statistics, Springer, New York (1999).
2[2] H. H. Barrett, K. J. Myers, Foundations of Image Science , John Wiley & Sons, Hoboken, NJ (2004).
3[3] H. Barrett, C. Abbey, E. Clarkson, "Objective assessment of image quality. III. ROC metrics, ideal observers, and likelihood-generating functions," J. Opt. Soc. Am. A 15, 1520-1535 (1998).
4[4] E. Clarkson , F. Shen, “Fisher information and surrogate figures of merit for the task-based assessment of image qualit y ,” JOSA A 27, 2313-2326 (2010) .
5[5] F. Shen, E. Clarkson, “Using Fisher information to approximate ideal observer performance on detection tasks for lumpy-background images,” JOSA A 23, 2406-2414 (2006).
6[6] E. Clarkson, "Asymptotic ideal observers and surrogate figures of merit for signal detection with list-mode data," J. Opt. Soc. Am. A 29, 2204-2216 (2012).
7[7] H. L. van Trees, Detection, Estimation and Modulation Theory , Part 1, New York, Wiley, (1968).
8[8] R.D. Gill, B.Y. Levit, “Applications of the van Trees inequality: a Bayesian Cramér–Rao bound, “ Bernoulli 1, 59–79 (1995).