Cost-Performance Tradeoffs in Fusing Unreliable Computational Units

TL;DR

This paper analyzes how to optimally fuse unreliable computational units to minimize cost while achieving a desired error performance, revealing different strategies based on cost functions.

Contribution

It provides a theoretical framework for understanding cost-performance tradeoffs in fusing unreliable units, with insights applicable to neuroscience, circuits, and crowdsourcing.

Findings

Optimal strategies vary with cost function shape.

Fusing multiple units is beneficial under convex costs.

Single reliable units are preferable under linear or concave costs.

Abstract

We investigate fusing several unreliable computational units that perform the same task. We model an unreliable computational outcome as an additive perturbation to its error-free result in terms of its fidelity and cost. We analyze performance of repetition-based strategies that distribute cost across several unreliable units and fuse their outcomes. When the cost is a convex function of fidelity, the optimal repetition-based strategy in terms of incurred cost while achieving a target mean-square error (MSE) performance may fuse several computational units. For concave and linear costs, a single more reliable unit incurs lower cost compared to fusion of several lower cost and less reliable units while achieving the same MSE performance. We show how our results give insight into problems from theoretical neuroscience, circuits, and crowdsourcing.

Equations

$$\displaystyle Y=f\mathopen{}\mathclose{{}\left(\mathbf{X}}\right),$$

$$Z_{\theta }=Y+U_{\theta },$$

$$\displaystyle\Pr\mathopen{}\mathclose{{}\left(\mathopen{}\mathclose{{}\left|Z_{\theta}-Y}\right|\geq\varepsilon}\right)\leq\frac{1}{\varepsilon^{2}\theta}.$$

$$\displaystyle C\mathopen{}\mathclose{{}\left(\theta}\right)=C_{\mathrm{min}}+G\mathopen{}\mathclose{{}\left(\theta}\right),$$

$$Z_{{\theta }_i}=Y+U_{{\theta }_i},$$

$$\displaystyle\hat{Y}_{N}\mathopen{}\mathclose{{}\left(\mathbf{w};\boldsymbol{\theta}}\right)$$

$$\displaystyle\sum_{i=1}^{N}C\mathopen{}\mathclose{{}\left(\theta_{i}}\right)+D\mathopen{}\mathclose{{}\left(N}\right),$$

$$\displaystyle\mathrm{MSE}\mathopen{}\mathclose{{}\left(\mathbf{w},\boldsymbol{\theta}}\right)\triangleq\operatorname{\mathbb{E}}\mathopen{}\mathclose{{}\left[\mathopen{}\mathclose{{}\left(\hat{Y}_{N}\mathopen{}\mathclose{{}\left(\mathbf{w};\boldsymbol{\theta}}\right)-Y}\right)^{2}}\right].$$

$$\displaystyle\mathrm{MSE}_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\boldsymbol{\theta}}\right)\triangleq\min\limits_{\mathbf{w}^{T}\mathbf{1}=1}\mathrm{MSE}\mathopen{}\mathclose{{}\left(\mathbf{w},\boldsymbol{\theta}}\right),$$

$$\displaystyle\mathrm{MSE}_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\boldsymbol{\theta}}\right)=\frac{1}{\boldsymbol{\theta}^{T}\mathbf{1}},\;\;\mathbf{w}_{\mathrm{o}}=\frac{\boldsymbol{\theta}}{\boldsymbol{\theta}^{T}\mathbf{1}},$$

$$\displaystyle\hat{Y}_{N}\mathopen{}\mathclose{{}\left(\mathbf{w}_{\mathrm{o}};\boldsymbol{\theta}}\right)=\mathbf{w}_{\mathrm{o}}^{T}\mathbf{Z}_{\scriptstyle\boldsymbol{\theta}}=\frac{\boldsymbol{\theta}^{T}\mathbf{Z}_{\scriptstyle\boldsymbol{\theta}}}{\boldsymbol{\theta}^{T}\mathbf{1}}.$$

$$\displaystyle\operatorname{\mathbb{E}}\mathopen{}\mathclose{{}\left[e^{\lambda U_{\theta}}}\right]\leq\exp\mathopen{}\mathclose{{}\left(\frac{\lambda^{2}}{2\theta}}\right),\;\;\forall\lambda\in\mathbb{R},$$

$$\displaystyle\Pr\mathopen{}\mathclose{{}\left(\mathopen{}\mathclose{{}\left|Z_{\theta}-Y}\right|\geq\varepsilon}\right)\leq 2\exp\mathopen{}\mathclose{{}\left(-{\varepsilon^{2}\theta/2}}\right).$$

$$\displaystyle\Pr\mathopen{}\mathclose{{}\left(\mathopen{}\mathclose{{}\left|\hat{Y}_{N}\mathopen{}\mathclose{{}\left(\mathbf{w};\boldsymbol{\theta}}\right)-Y}\right|\geq\varepsilon}\right)\leq 2\exp\mathopen{}\mathclose{{}\left(-\frac{\varepsilon^{2}}{\sum_{i=1}^{N}w_{i}^{2}/\theta_{i}}}\right).$$

$$\displaystyle\Pr\mathopen{}\mathclose{{}\left(\mathopen{}\mathclose{{}\left|\hat{Y}_{N}\mathopen{}\mathclose{{}\left(\mathbf{w}_{\mathrm{o}};\boldsymbol{\theta}}\right)-Y}\right|\geq\varepsilon}\right)\leq 2\exp\mathopen{}\mathclose{{}\left(-{\varepsilon^{2}\boldsymbol{\theta}^{T}\mathbf{1}/2}}\right),$$

$$\displaystyle\tau=\mathrm{MSE}_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\boldsymbol{\theta}}\right)=\frac{1}{\boldsymbol{\theta}^{T}\mathbf{1}},$$

$$\displaystyle\mathrm{Cost}_{\tau}\mathopen{}\mathclose{{}\left(N}\right)\triangleq\sum_{i=1}^{N}C\mathopen{}\mathclose{{}\left(\theta_{i}}\right)+D\mathopen{}\mathclose{{}\left(N}\right).$$

$${\theta }_i\triangleq \frac{1}{\tau N},i=1,\dots ,N,$$

$$\displaystyle\min_{\boldsymbol{\theta}\in\mathbb{R}^{N}_{+}}\sum_{i=1}^{N}C\mathopen{}\mathclose{{}\left(\theta_{i}}\right)$$

$$\displaystyle\mathrm{Cost}_{\tau}\mathopen{}\mathclose{{}\left(N}\right)$$

$$\displaystyle=NG\mathopen{}\mathclose{{}\left(\frac{1}{\tau N}}\right)+NC_{\mathrm{min}}+D\mathopen{}\mathclose{{}\left(N}\right).$$

$${\mathrm{Cost}}_{\tau }$$

$$\displaystyle\mathrm{Cost}_{\tau}\mathopen{}\mathclose{{}\left(a}\right)$$

$$\displaystyle N_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\tau}\right)=\operatornamewithlimits{arg\ min}\limits_{N\in\mathopen{}\mathclose{{}\left\{\lfloor a_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\tau}\right)\rfloor,\lceil a_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\tau}\right)\rceil}\right\}}\mathrm{Cost}_{\tau}\mathopen{}\mathclose{{}\left(N}\right).$$

$$\displaystyle C_{\mathrm{min}}+D^{\prime}\mathopen{}\mathclose{{}\left(1}\right)<V\mathopen{}\mathclose{{}\left(\tau}\right)$$

$$\displaystyle V\mathopen{}\mathclose{{}\left(\tau}\right)\triangleq\tau^{-1}G^{\prime}\mathopen{}\mathclose{{}\left(\tau^{-1}}\right)-G\mathopen{}\mathclose{{}\left(\tau^{-1}}\right).$$

$$\displaystyle\kappa_{\tau}\mathopen{}\mathclose{{}\left(1}\right)=G\mathopen{}\mathclose{{}\left(\tau^{-1}}\right)-\tau^{-1}G^{\prime}\mathopen{}\mathclose{{}\left(\tau^{-1}}\right)+C_{\mathrm{min}}+D^{\prime}\mathopen{}\mathclose{{}\left(a}\right)<0$$

$$\displaystyle\frac{\partial\mathrm{Cost}_{\tau}\mathopen{}\mathclose{{}\left(a}\right)}{\partial a}\Bigg{\rvert}_{a=a_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\tau}\right)}$$

$$=0.$$

$$\displaystyle L\triangleq\lim_{\tau\rightarrow 0}V\mathopen{}\mathclose{{}\left(\tau}\right)>0,$$

$$\displaystyle T\triangleq\inf V^{-1}\mathopen{}\mathclose{{}\left(C_{\mathrm{min}}+D^{\prime}\mathopen{}\mathclose{{}\left(1}\right)}\right)>0,$$

$$\displaystyle C\mathopen{}\mathclose{{}\left(\theta}\right)=C_{\mathrm{min}}+\alpha\theta,\;\;\;\theta>0,$$

Full text

1 Introduction

We consider the problem of fusing outcomes of several unreliable computational units that perform the same computation under cost and fidelity constraints. We formalize the relationship between the fidelity of each unit and the cost associated with it, and explore this tradeoff in a number of practical problems. Consider, for instance, the capacity of an additive white Gaussian noise (AWGN) channel, which is a logarithmic function of the signal-to-noise (SNR) ratio. In this scenario, the capacity can be increased at the expense of requiring a higher SNR, which introduces a tradeoff between cost (SNR) and performance (rate). Note also that the Fisher information in estimation is often a linear function of SNR, leading to a different cost-performance tradeoff [2].

Building reliable systems out of unreliable components has attracted substantial interest in circuits and systems [3, 4, 5], information theory [6, 7, 8], and signal processing [9]. In [3], Von Neumann investigated error in logic circuits from a statistical point of view and demonstrated that repeated computations followed by majority logic may yield reliable results even when the underlying components are unreliable. In [4], Tryon introduced a technique called quadded logic, which corrects errors by a redundant design of logic gates. Moreover, the authors of [6, 7, 8] investigated reliable computation by formulas in the presence of noise. More recently, the authors of [9] considered energy-reliability tradeoffs in computing linear transforms implemented on unreliable components.

Fusion of the outputs collected from several sensors has been considered in distributed detection, estimation, classification, and optimization in sensor networks [10, 11, 12, 13, 14, 15, 16]. Often, spatially distributed sensors locally perform a decision-making task and send their outputs, under bandwidth constraints, to a fusion center that forms a final decision. In most practical applications, these sensors are battery-powered devices with limited accuracy and computational capabilities, so their performance is critically affected by the resources allocated to them, introducing a cost-performance tradeoff. The authors of [15] studied tradeoffs between the number of sensors, resolution of quantization at each sensor, and SNR. Similarly, [16] considered the tradeoff between reliability and efficiency in distributed source coding for field-gathering sensor networks. In general, the main goal is to make a reliable final decision in a cost-efficient manner based on these unreliable sensors subject to resource and reliability constraints.

A fundamental question that arises in fusing several unreliable computational units is how a limited budget should be allocated across several unreliable units, where adding a new unit incurs a baseline cost as well as an incremental cost, and also increases the cost of fusion. That is, what is the optimal approach in terms of cost-performance tradeoff? Although existing work in fault-tolerant computing and in-sensor networks focus on different pieces of this problem, a more general treatment that jointly considers cost and performance is necessary. This paper is an attempt to combine insights from both fields into a unified framework that captures characteristics of a range of problems. In particular, we show how our framework and results are connected to problems from neuroscience, circuits, and crowdsourcing in Section 5.

In this paper, we present an abstract framework to explore the fundamental tradeoff between cost and performance achievable through forms of redundancy. We model unreliability in any computational unit as an additive random perturbation, where the variance of the perturbation is inversely related to its fidelity. We cast the main task as inference of the error-free computation based on noisy computational outcomes. Each computational unit incurs some cost, which is a function of its fidelity, that includes a baseline cost incurred simply to operate the unit.

We define a class of repetition-based strategies, where each strategy distributes the total cost across several unreliable computational units and fuses their outputs. We note that the fusion operation also incurs some cost, which is a function of the number of individual computational units to be fused. We measure the inference performance of each strategy in terms of MSE between its final output and the error-free computation.

We consider optimal repetition-based strategies under convex, linear, and concave cost functions rather than restricting to specific cost functions. For convex costs, there are two main cases. In the first case, we prove that using only a single and more reliable computational outcome is more cost-efficient than the fusion of several lower cost but less reliable computational outcomes. In the second case, however, we demonstrate that the optimal strategy uses several computational outcomes instead of a single more reliable one. Intuitively, the convexity of the cost function disperses the cost across several less reliable computational outcomes with smaller individual costs. For linear or concave costs, the optimal strategy is to use a single and more reliable computational outcome.

2 Problem Description

Consider the problem of fusing outcomes of several unreliable computational units subject to cost and fidelity constraints. We first introduce a model of an unreliable computational outcome as an additive perturbation to its error-free result in terms of its fidelity and cost. We next consider a class of repetition-based strategies that distribute cost across several parallel unreliable units and fuse their outcomes to produce a final estimate of the error-free computation.

Suppose a vector of input signals \mathbf{X}=\mathopen{}\mathclose{{}\left(X_{1},\ldots,X_{k}}\right) is processed to yield the error-free computation,

[TABLE]

where f\mathopen{}\mathclose{{}\left(\cdot}\right) is some arbitrary target function. Instead, we observe an unreliable computational outcome,

[TABLE]

where $U_{\theta}$ is a zero-mean perturbation with variance $\theta^{-1}$. Here, $\theta$ is the fidelity of the unreliable computational outcome $Z_{\theta}$. We assume that $Y$ and $U_{\theta}$ are uncorrelated, that is, \operatorname{\mathbb{E}}\mathopen{}\mathclose{{}\left[YU_{\theta}}\right]=\operatorname{\mathbb{E}}\mathopen{}\mathclose{{}\left[Y}\right]\operatorname{\mathbb{E}}\mathopen{}\mathclose{{}\left[U_{\theta}}\right] holds, whether or not $Y$ is a random variable.

By Chebyshev’s inequality, the unreliable outcome $Z_{\theta}$ with fidelity $\theta>0$ satisfies, for any $\varepsilon>0$,

[TABLE]

This implies the unreliable outcome $Z_{\theta}$ converges to the error-free computation in probability as the fidelity tends to infinity. However, as the fidelity parameter $\theta$ increases, the cost $C(\theta)$ incurred to guarantee that level of fidelity also increases, introducing a cost-fidelity tradeoff. Note that this holds both when $X_{i}$ for $i=1,\ldots,k$, or $Y$, are random as well as when they are purely deterministic.

In this model, we must incur a cost C\mathopen{}\mathclose{{}\left(\theta}\right) to get the unreliable outcome $Z_{\theta}$ with fidelity $\theta>0$, which we assume to be a strictly increasing function of $\theta$. In particular, we assume

[TABLE]

where C_{\mathrm{min}}\triangleq\inf_{\theta>0}C\mathopen{}\mathclose{{}\left(\theta}\right)\geq 0 is the minimum (baseline) cost, and G\mathopen{}\mathclose{{}\left(\theta}\right) is an increasing and twice differentiable incremental cost function with $G(0)=0$. In the sequel, we focus on three classes of cost functions: convex, linear, and concave.

We define a class of repetition-based strategies that fuse the outputs of several computational units to estimate $Y$. For any positive integer $N$, a repetition-based strategy $S_{N}$, with weights \mathbf{w}=\mathopen{}\mathclose{{}\left(w_{1},\ldots,w_{N}}\right)\in\mathbb{R}^{N} and fidelities \boldsymbol{\theta}=\mathopen{}\mathclose{{}\left(\theta_{1},\ldots,\theta_{N}}\right)\in\mathopen{}\mathclose{{}\left(0,\infty}\right)^{N}, linearly combines the outcomes of $N$ parallel unreliable units with fidelities $\boldsymbol{\theta}$ using the weights $\mathbf{w}$. That is, if we denote each unreliable outcome with the fidelity $\theta_{i}$ and the cost C\mathopen{}\mathclose{{}\left(\theta_{i}}\right) as

[TABLE]

for $i=1,\ldots,N$, then the final output of this strategy $S_{N}$ is

[TABLE]

where \mathbf{Z}_{\scriptstyle\boldsymbol{\theta}}\triangleq\mathopen{}\mathclose{{}\left(Z_{\theta_{1}},\ldots,Z_{\theta_{N}}}\right), \mathbf{U}_{\scriptstyle\boldsymbol{\theta}}\triangleq\mathopen{}\mathclose{{}\left(U_{\theta_{1}},\ldots,U_{\theta_{N}}}\right), and \mathbf{1}=\mathopen{}\mathclose{{}\left(1,\ldots,1}\right)\in\mathbb{R}^{N} is a vector of ones. In particular, we assume that $U_{\theta_{i}}$s are uncorrelated with each other.

The cost incurred by the strategy $S_{N}$ with fidelities $\boldsymbol{\theta}$ is

[TABLE]

where D\mathopen{}\mathclose{{}\left(N}\right) is the fusion cost, i.e., the cost of linear combination. We assume that the function $D:\mathbb{Z}_{+}\rightarrow\mathbb{R}_{+}$ is increasing, as fusing a larger number of computational units has higher cost than fewer. Note that the fusion cost is super-linear in $N$ in that it requires at least O\mathopen{}\mathclose{{}\left(N}\right) multiplications and additions. In particular, we assume that D\mathopen{}\mathclose{{}\left(N}\right) is convex in $N$.

3 Performance Analysis

Here, we consider the MSE performance of each repetition-based strategy in estimating the error-free computation $Y$. For any positive integer $N$, the strategy $S_{N}$ with a weight vector $\mathbf{w}\in\mathbb{R}^{N}$ and a fidelity vector \boldsymbol{\theta}\in\mathopen{}\mathclose{{}\left(0,\infty}\right)^{N} achieves the MSE

[TABLE]

In particular, we derive the minimum MSE (MMSE) achievable by this strategy $S_{N}$ while producing an unbiased output:

[TABLE]

where $\mathbf{w}_{\mathrm{o}}$ is the corresponding minimizer.

Lemma 1**.**

Suppose that for any positive integer $N$, the strategy $S_{N}$ fuses the outcomes of $N$ parallel computational units with fidelities \boldsymbol{\theta}\in\mathopen{}\mathclose{{}\left(0,\infty}\right)^{N}. Then the MMSE achievable by this strategy $S_{N}$ while producing an unbiased estimate of $Y$, and the corresponding weights are

[TABLE]

respectively.

Proof.

We provide the proof in Appendix A. ∎

Thus, Lemma 1 provides the strategy $S_{N}$ achieving the MMSE for a given fidelity vector \boldsymbol{\theta}\in\mathopen{}\mathclose{{}\left(0,\infty}\right)^{N}. For any positive integer $N$, whenever we refer to the strategy $S_{N}$, we use the optimal weights given in (4), so that its output is

[TABLE]

We next study a particular scenario, where $U_{\theta}$ is sub-Gaussian.

3.1 Sub-Gaussian Perturbations

Here, we consider a case where the perturbation $U_{\theta}$ is sub-Gaussian with parameter $\theta^{-1}$, which means [17]

[TABLE]

or equivalently, the probability of absolute deviation of $Z_{\theta}$ from $Y$ satisfies, for any $\varepsilon>0$,

[TABLE]

The tail bound in (6) decreases faster (with increasing $\theta$) than the bound in (1). Sub-Gaussian distributions can be used to model a wide range of stochastic phenomena including Gaussian and uniform distributions, or distributions with finite or bounded support. Note that a weighted sum of finitely many sub-Gaussian random variables is also sub-Gaussian [17]. By appling this result to the output of a strategy $S_{N}$ with $\mathbf{w}\in\mathbb{R}^{N}$ and \boldsymbol{\theta}\in\mathopen{}\mathclose{{}\left(0,\infty}\right)^{N}, we get, for any $\varepsilon>0$,

[TABLE]

The weights minimizing the upper bound under $\mathbf{w}^{T}\mathbf{1}=1$, and the resulting bound are known to be $\mathbf{w}_{\mathrm{o}}=\boldsymbol{\theta}/\boldsymbol{\theta}^{T}\mathbf{1},$ and

[TABLE]

for any $\varepsilon>0$, respectively.

We emphasize that, in this case, even though the performance is measured in terms of probability of absolute deviation from the error-free computation, the optimal weights are exactly the same as the ones minimizing the MSE. Hence, same results apply to both cases when comparing the cost-performance tradeoff of the repetition-based strategies.

In this section, we analyzed the MSE performance of repetition-based strategies. More precisely, for any positive integer $N$ and a fidelity vector \boldsymbol{\theta}\in\mathopen{}\mathclose{{}\left(0,\infty}\right)^{N}, we derived the optimal weights for the strategy $S_{N}$ in terms of minimizing the MSE. Based on these results, we next investigate the cost-performance tradeoff for the class of repetition-based strategies under the classes of convex, linear, and concave cost functions.

4 Cost-Performance Tradeoff

We investigate the performance of repetition-based strategies under convex, linear, and concave cost functions in terms of the tradeoff between the total incurred cost and the final MSE performance in estimating the error-free computation.

We first analyze the case where the cost C\mathopen{}\mathclose{{}\left(\theta}\right) is a convex function of the fidelity $\theta$. We characterize the optimal strategy, based on the desired MSE performance as well as the baseline and fusion cost functions. In particular, we show that the optimal cost-performance tradeoff may be achieved by some strategy $S_{N}$ with $N>1$ under certain conditions.

We next consider the case where the cost C\mathopen{}\mathclose{{}\left(\theta}\right) is a linear function of the fidelity parameter $\theta$, and show that strategy $S_{1}$ is optimal among repetition-based strategies. We finally study the concave cost scenario, and demonstrate results similar to the linear cost function case.

To compare cost-performance tradeoffs of repetition-based strategies, we constrain each strategy to guarantee the same MSE performance. More precisely, given some $\tau>0$, we assume that the strategy $S_{N}$ with \boldsymbol{\theta}\in\mathopen{}\mathclose{{}\left(0,\infty}\right)^{N} satisfies

[TABLE]

or equivalently, $\tau^{-1}=\boldsymbol{\theta}^{T}\mathbf{1}$, for any positive integer $N$. We also define the total cost incurred by this strategy $S_{N}$, which achieves \mathrm{MSE}_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\boldsymbol{\theta}}\right)=\tau, as

[TABLE]

4.1 Convex Cost Functions

We study the cost-performance tradeoff for the class of repetition-based strategies under a convex cost function. This case corresponds to a law of diminishing returns between cost and fidelity, which may drive the dispersion of cost across several less reliable computational units with smaller individual costs. We show that there are two main cases, where in the first case some strategy $S_{N}$ with $N>1$ may incur the minimum total cost achievable by the repetition-based strategies while achieving the same MSE, whereas in the second case the strategy $S_{1}$ is optimal in terms of cost-performance tradeoff, i.e., no repetition or fusion is required.

Consider a uniform fidelity distribution across several unreliable computational outcomes, given by

[TABLE]

which implies that the constraint \mathrm{MSE}_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\boldsymbol{\theta}}\right)=\tau is satisfied. In fact, the following lemma shows that the optimal fidelity distribution satisfying the MSE constraint in terms of minimizing the total cost is in fact uniform.

Lemma 2**.**

For any $\tau>0$, the uniform fidelity distribution given by (7) is the unique solution to the optimization problem:

[TABLE]

subject to $\boldsymbol{\theta}^{T}\mathbf{1}=\tau^{-1}$ when the cost function C\mathopen{}\mathclose{{}\left(\theta}\right) is convex.

Proof.

The proof is given in Appendix B. ∎

Hence, we only consider the case where the strategy $S_{N}$, for each positive integer $N$, uses the fidelities in (7). The total cost incurred by this strategy $S_{N}$ is:

[TABLE]

To investigate the behavior of the total cost, we define its continuous relaxation as

[TABLE]

where D\mathopen{}\mathclose{{}\left(a}\right) is a twice differentiable continuous relaxation of the fusion cost function D\mathopen{}\mathclose{{}\left(N}\right). We first demonstrate that \mathrm{Cost}_{\tau}\mathopen{}\mathclose{{}\left(a}\right) is a convex function in $a$.

Lemma 3**.**

The total cost function \mathrm{Cost}_{\tau}\mathopen{}\mathclose{{}\left(a}\right) is convex in $a$.

Proof.

The proof is provided in Appendix C. ∎

Convexity of \mathrm{Cost}_{\tau}\mathopen{}\mathclose{{}\left(a}\right) implies that it has a unique minimizer on any given compact subset of its domain \mathopen{}\mathclose{{}\left[1,\infty}\right). In particular, note that \mathrm{Cost}_{\tau}\mathopen{}\mathclose{{}\left(1}\right)=G\mathopen{}\mathclose{{}\left(\tau^{-1}}\right)+C_{\mathrm{min}}, and \mathrm{Cost}_{\tau}\mathopen{}\mathclose{{}\left(a}\right)\rightarrow\infty as $a\rightarrow\infty$. Therefore, the total cost function \mathrm{Cost}_{\tau}\mathopen{}\mathclose{{}\left(a}\right) has a unique and finite minimizer a_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\tau}\right)\in\mathopen{}\mathclose{{}\left[1,\infty}\right). Also, there exists a corresponding unique optimal repetition-based strategy, which we denote as the strategy S_{N_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\tau}\right)} where

[TABLE]

is a finite positive integer (a function of $\tau$), that minimizes the total incurred cost while achieving the desired MSE of $\tau>0$.

We next characterize conditions under which the optimal repetition-based strategy either uses a single but more reliable computational unit, that is, N_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\tau}\right)=1, or distributes the cost across several unreliable computational units and fuses their outcomes, that is, N_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\tau}\right)>1. In the latter case, we implicitly derive the optimal strategy as a function of the desired MSE level $\tau$, the baseline cost $C_{\mathrm{min}}$, and the fusion cost function D\mathopen{}\mathclose{{}\left(\cdot}\right). The next theorem characterizes these cases in terms of the first derivative of the fusion cost and the baseline cost.

Theorem* 1***.**

For any given $\tau>0$, the minimizer of \mathrm{Cost}_{\tau}\mathopen{}\mathclose{{}\left(a}\right) satisfies a_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\tau}\right)>1 if and only if

[TABLE]

where

[TABLE]

Proof.

We define \kappa_{\tau}\mathopen{}\mathclose{{}\left(a}\right)\triangleq\partial\mathrm{Cost}_{\tau}\mathopen{}\mathclose{{}\left(a}\right)/\partial a, and observe that from Lemma 3, \kappa_{\tau}\mathopen{}\mathclose{{}\left(a}\right) is nondecreasing and continuous in $a$ since \mathrm{Cost}_{\tau}\mathopen{}\mathclose{{}\left(a}\right) is a twice differentiable and convex function of $a$. Hence, whenever \kappa_{\tau}\mathopen{}\mathclose{{}\left(1}\right)\geq 0, we have \kappa_{\tau}\mathopen{}\mathclose{{}\left(a}\right)\geq 0 for any $a>1$. It implies that \mathrm{Cost}_{\tau}\mathopen{}\mathclose{{}\left(a}\right) is a nondecreasing function of $a$ on \mathopen{}\mathclose{{}\left[1,\infty}\right), and minimized at a_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\tau}\right)=1. When \kappa_{\tau}\mathopen{}\mathclose{{}\left(1}\right)<0, \mathrm{Cost}_{\tau}\mathopen{}\mathclose{{}\left(a}\right) is minimized at some finite a_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\tau}\right)>1, since \mathrm{Cost}_{\tau}\mathopen{}\mathclose{{}\left(a}\right)\rightarrow\infty as $a\rightarrow\infty$. The proof follows by noting that

[TABLE]

if and only if C_{\mathrm{min}}+D^{\prime}\mathopen{}\mathclose{{}\left(1}\right)<V\mathopen{}\mathclose{{}\left(\tau}\right), where V\mathopen{}\mathclose{{}\left(\tau}\right) is defined in (10). ∎

Based on these results, we can characterize the optimal repetition-based strategy. If C_{\mathrm{min}}+D^{\prime}\mathopen{}\mathclose{{}\left(1}\right)\geq V\mathopen{}\mathclose{{}\left(\tau}\right), then N_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\tau}\right)=1 since a_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\tau}\right)=1. Otherwise, we get a_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\tau}\right)>1, which is in this case implicitly given by

[TABLE]

If 1<a_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\tau}\right)<2, then we may get N_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\tau}\right)=1 or N_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\tau}\right)=2, based on (9). When a_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\tau}\right)\geq 2, we get N_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\tau}\right)>1.

We finally consider the optimal repetition-based strategy as the target MSE $\tau$ changes. In the following lemma, we investigate the function V\mathopen{}\mathclose{{}\left(\tau}\right) defined in (10) as $\tau$ changes.

Lemma 4**.**

V\mathopen{}\mathclose{{}\left(\tau}\right)* is nonnegative and nonincreasing on $(0,\infty)$, and in particular, we have \lim_{\tau\rightarrow\infty}V\mathopen{}\mathclose{{}\left(\tau}\right)=0, and*

[TABLE]

if V\mathopen{}\mathclose{{}\left(\tau}\right) is bounded as $\tau\rightarrow 0$, or else, the limit does not exist.

Proof.

We present the proof in Appendix D. ∎

It may appear that from (9) and (11), as the target MSE $\tau$ decreases, the optimal repetition-based strategy may need to fuse more units, i.e., N_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\tau}\right) may increase. More rigorously, we next characterize the behavior of the minimizer a_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\tau}\right) of the total cost \mathrm{Cost}_{\tau}\mathopen{}\mathclose{{}\left(a}\right) as the target MSE $\tau$ changes.

Theorem* 2***.**

If the limit in (12) exists, and L\leq C_{\mathrm{min}}+D^{\prime}\mathopen{}\mathclose{{}\left(1}\right), then a_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\tau}\right)=1 for all $\tau>0$. If, on the other hand, the limit does not exist, or it exists and L>C_{\mathrm{min}}+D^{\prime}\mathopen{}\mathclose{{}\left(1}\right), we define

[TABLE]

where V^{-1}\mathopen{}\mathclose{{}\left(x}\right) is the inverse image of a point $x$ under the function $V$ for any $x>0$. Then we get a_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\tau}\right)=1 whenever $\tau\geq T$, and a_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\tau}\right)>1 whenever $0<\tau<T$.

Proof.

Suppose the limit in (12) exists, and L\leq C_{\mathrm{min}}+D^{\prime}\mathopen{}\mathclose{{}\left(1}\right). Then V\mathopen{}\mathclose{{}\left(\tau}\right)\leq C_{\mathrm{min}}+D^{\prime}\mathopen{}\mathclose{{}\left(1}\right), and a_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\tau}\right)=1, for all $\tau>0$.

Suppose next the limit in (12) either does not exist, or it exists and L>C_{\mathrm{min}}+D^{\prime}\mathopen{}\mathclose{{}\left(1}\right). Since V\mathopen{}\mathclose{{}\left(\tau}\right) is a monotone function, V^{-1}\mathopen{}\mathclose{{}\left(C_{\mathrm{min}}+D^{\prime}\mathopen{}\mathclose{{}\left(1}\right)}\right) is either a singleton or an interval. Then for any $\tau\geq T$, we have V\mathopen{}\mathclose{{}\left(\tau}\right)\leq C_{\mathrm{min}}+D^{\prime}\mathopen{}\mathclose{{}\left(1}\right), which implies a_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\tau}\right)=1, and when $0<\tau<T$, we have C_{\mathrm{min}}+D^{\prime}\mathopen{}\mathclose{{}\left(1}\right)<V\mathopen{}\mathclose{{}\left(\tau}\right), which implies a_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\tau}\right)>1. ∎

In this section, we investigated the cost-performance tradeoff for repetition-based strategies under convex cost functions. In particular, we characterized the optimal repetition-based strategy in terms of the baseline cost, the behaviors of the incremental and fusion cost functions with different parameters, for different values of the target MSE level $\tau$. We next study the cost-performance tradeoff under linear cost functions.

4.2 Linear Cost Functions

We consider the optimal repetition-based strategy in terms of cost-efficiency when the underlying cost function is linear, where we can express it as

[TABLE]

where $\alpha>0$ is an application-dependent constant. This case corresponds to a law of proportional returns. We show that the strategy $S_{1}$ is the optimal repetition-based strategy for any target MSE $\tau>0$. There is no gain in repetition-based approaches in terms of cost-efficiency for linear cost functions.

Theorem* 3***.**

Suppose that the cost function C\mathopen{}\mathclose{{}\left(\theta}\right) is linear, that is, C\mathopen{}\mathclose{{}\left(\theta}\right)=C_{\mathrm{min}}+\alpha\theta for some $\alpha>0$. Then the optimal repetition-based strategy in terms of minimizing the incurred cost while achieving the same MSE is the strategy $S_{1}$.

Proof.

Let $\tau>0$ be given. The total cost of the strategy $S_{N}$, for any positive integer $N$, is given by

[TABLE]

This implies the cost incurred by the strategy $S_{1}$ is smaller than that of the strategy $S_{N}$ for any $N>1$ and $\tau>0$. ∎

For proportional costs a single more reliable unit is always more cost-efficient than a fusion of several less reliable units in the sense that it incurs a smaller cost while achieving the same MSE. We next analyze the concave cost function case.

4.3 Concave Cost Functions

We consider the cost-performance tradeoff of each strategy in the class of strategies when the cost function is concave. This case corresponds to a law of increasing returns, as opposed to a law of diminishing returns. That is, the incremental cost for performance decreases, making single, high-cost, high performance elements more attractive. Before proving the main theorem of this section, we present a lemma that proves that the concave incremental cost function is sub-additive.

Lemma 5**.**

If a function $f$ with the domain \mathopen{}\mathclose{{}\left[0,\infty}\right) is concave, and $f(0)\geq 0$, then it is sub-additive, i.e., for any $x,y\geq 0$,

[TABLE]

Proof.

We provide the proof in Appendix E. ∎

The next theorem characterizes the optimal repetition-based strategy in terms of minimizing the total incurred cost while achieving the same MSE performance for a given $\tau>0$.

Theorem* 4***.**

Suppose that the cost function C\mathopen{}\mathclose{{}\left(\theta}\right) is concave, and each repetition-based strategy achieves the same MSE level $\tau>0$. Then the strategy $S_{1}$ is always the optimal strategy in terms of incurring the smallest cost for any $\tau>0$.

Proof.

Let $\tau>0$ be given. Then, for any positive integer $N$, the total cost incurred by the strategy $S_{N}$ is given by

[TABLE]

We note that by Lemma 5, the incremental cost function is sub-additive, since it is concave and G\mathopen{}\mathclose{{}\left(0}\right)\geq 0, implying that

[TABLE]

Note that the cost incurred by the strategy $S_{1}$ is given by

[TABLE]

implying \mathrm{Cost}_{\tau}\mathopen{}\mathclose{{}\left(N}\right)>\mathrm{Cost}_{\tau}\mathopen{}\mathclose{{}\left(1}\right) for any $N>1$. Hence, the strategy $S_{1}$ is the optimal strategy for any desired MSE. ∎

Strategy $S_{1}$, which is formed by exhausting all available cost for a single computational unit, is more cost-efficient as compared to any strategy $S_{N}$ with $N>1$, which allocates available cost across several less reliable computational units.

In this section, we considered the cost-performance tradeoff of repetition-based strategies under convex, linear, and concave cost function classes. We showed that under convex cost functions the optimal cost-performance tradeoff may be achieved either by the strategy $S_{1}$ or by some strategy $S_{N}$ with $N>1$ under certain conditions. For linear and concave costs, optimality is always achieved by strategy $S_{1}$ for any target MSE performance. In the next section, we consider applications of our results into a number of contexts.

5 Applications

Here, we show how our cost-fidelity formulation and theoretical results are connected to problems from different fields.

5.1 Neuroscience

We review a particular application of our framework in a theoretical neuroscience context. We focus on two principal tasks of the brain where synapses play essential roles, namely, information storage and information processing. Typical central synapses exhibit noisy behavior due, for instance, to probabilistic transmitter release. The firing of the presynaptic neuron is inherently stochastic and occasionally fails to evoke an excitatory postsynaptic potential (EPSP). In this sense, we can cast each noisy synapse as an unreliable computational unit, contributing to the overall neural computation carried out by its efferent neuron. We focus on two distinct cost-fidelity formulations, where we show that experimental results [18, 19] agree with our theoretical predictions. We note that recall corresponds to a form of “in-memory computing” whereas processing corresponds to a form of “in-sensor computing”.

5.1.1 In-Memory Computing

Revisiting [18], we first consider an information-theoretic framework to study the information storage capacity of synapses under resource constraints, where memory is seen as a communication channel subject to several sources of noise. Each synapse has a certain SNR, where increasing the SNR increases the information storage capacity in a logarithmic fashion. However, this increase comes at a cost, namely, the synaptic volume. Hence, from an information storage perspective, we cast capacity as the fidelity of a noisy synapse and the volume as the cost. If we denote the information storage capacity of a synapse and its average volume by $C_{\mathrm{I}}$ and $V$, respectively, then taking Shannon’s AWGN channel capacity formula [20] for concreteness:

[TABLE]

where $V_{N}$ is the volume of a synapse with a unit SNR. This relationship assumes the power law (V/V_{N})=\mathopen{}\mathclose{{}\left(A/A_{N}}\right)^{2}, which is supported by experimental measurements [18], where $A$ is the mean EPSP amplitude and $A_{N}$ is the noise amplitude. We rewrite the volume as a function of capacity as

[TABLE]

and observe that this is an exponential cost function, a particular example of convex costs. For exponential costs, fusion of several less reliable computational units may lead to better cost-efficiency than a single more reliable computational unit. Therefore, our cost-fidelity framework applied to information recall under resource constraints recovers the principle that several small and noisy synapses should be present in brain regions performing storage and recall, rather than large and isolated synapses [18, 21].

Moreover, [22, 23, 24, 25, 26] show that the noisiness of the synapses leads to efficient information transmission. That is, transmitting the same information over several less reliable but metabolically cheaper synapses requires less energy, as compared to the case where the information is transmitted over a single, more reliable but metabolically more expensive synapse. The idea that noise can facilitate information transmission is also present in neuronal networks. In particular, the authors in [27] show that a neuron is a noise-limited device of restricted bandwidth, and an energy-efficient nervous system will split the information and transmit it over a large number of relatively noisy neurons of lower information capacity.

5.1.2 In-Sensor Computing

We next consider an information processing perspective, and view the SNR of a synapse itself as its fidelity and the synaptic volume as the cost. We adopt a data-driven approach using two different data sets. This joining is necessary since joint electrophysiology and imaging experiments are technically difficult, where electrophysiology experiments to measure voltages require live tissue while electron micrograph imaging experiments to measure volumes require fixing and slicing the tissue [19].

The first data set [18] includes EPSP measurements across 637 distinct synapses over 43 trials for each synapse. Based on these measurements, we generate an empirical distribution of the mean EPSP measurements of a synapse. The second data set [19] includes volume measurements across 357 synapses, which is used to compute a distribution of a synapse volume.

We first generate $T=500$ random variables \mathopen{}\mathclose{{}\left\{Y_{t}}\right\}_{t=1}^{T} from the calculated volume distribution. We next generate $T$ random variables from the calculated mean EPSP distribution, and sort them assuming a monotonic relationship between the mean EPSP and the volume of synapses [18]. From the sorted mean EPSP amplitudes, we compute the corresponding SNRs \mathopen{}\mathclose{{}\left\{X_{t}}\right\}_{t=1}^{T}. We plot the resulting pairs \mathopen{}\mathclose{{}\left\{\mathopen{}\mathclose{{}\left(X_{t},Y_{t}}\right)}\right\}_{t=1}^{T} in Figure 1. This plot indicates that the cost function is approximately concave as a function of SNR. More rigorously, we assess convexity using a nonparametric hypothesis test based on a simplex statistic, a descriptive measure of curvature described in [28]. When applied to this data, the test yields a $p$-value of $3.25\times 10^{-4}$, which can be interpreted as a strong evidence in favor of the hypothesis that the cost (volume) is a concave function of the SNR (fidelity). This suggests that the brain may achieve cost-efficiency by using a single large and reliable synapse, instead of several smaller and less reliable synapses, from an information processing perspective.

To compare this prediction with experimental findings, we focus on a particular synapse called the calyx of Held, the largest synapse in the mammalian auditory central nervous system that connects principal neurons within the auditory system [29, 30, 31]. The calyx of Held plays a crucial role in certain information processing tasks of the brain. For instance, the principal cells connected by the calyx of Held enable interaural level detection, a vital role in high frequency sound localization [32, 33]. The signals derived from the calyx of Held generate large excitatory postsynaptic currents with a short synaptic delay, where the transmission speed and fidelity of the calyx is very reliable in mature animals [34].

Hence, the calyx of Held may be regarded as a very reliable but costly synapse, as compared to the ones performing information storage tasks, which are noisier and less costly in terms of brain resources. We observe that these experimental findings agree with our prediction that the cost-efficiency results from employing a single reliable and costly synapse (calyx of Held), instead of several less reliable and metabolically cheaper synapses, under a concave cost function.

5.2 Circuits

Next, let us consider signal processing systems implemented on unreliable circuit fabrics. As CMOS technology scales beyond $10\text{\,}\mathrm{nm}$, the operation of CMOS devices begins to suffer from static defects as well as dynamic operational non-determinism [35, 36, 37]. Moreover, spintronics, which use electron spin for computing, exhibit an unreliable behavior, where there is a tradeoff between reliability and energy consumption [38, 39]. That is, probability of failure is smaller when more energy is used. Hence, deeply scaled CMOS and spintronics based systems must operate under computational errors.

In [3], von Neumann studied noise in circuits and showed that even when circuit components are unreliable, reliable computations can be performed by using repetition-based schemes. Repeated computations followed by a majority vote have also been used extensively in error-tolerant circuit design [40, 41]. Also, Hadjicostis [42] investigated redundancy-based approaches to build fault-tolerant dynamical systems out of cheap but unreliable components.

Moreover, a statistical error compensation technique called Algorithmic Noise Tolerance (ANT) has been studied in [43, 44]. ANT compensates for errors in computation in a statistical manner by fusing outcomes of several unreliable computational branches that operate at different points along energy-reliability tradeoffs. The ANT framework can also be cast as a CEO problem in multiterminal source coding [45].

Stochastic behavior in circuit fabrics may arise when computation is embedded into either memory, which leads to in-memory computing [46], or sensing, which leads to in-sensor computing [47], to achieve cost-efficiency [48]. Note that in-memory computing and in-sensor computing may lead to fundamentally different cost-performance tradeoffs. In particular, we demonstrate that the difference between in-memory computing and in-sensor computing may be modeled through our framework by using different cost-fidelity function classes.

5.2.1 Example case

Here, we present an application of the results of this section into spintronics. In particular, exponential cost has been shown to approximately model the functional dependence between energy and reliability for a typical spin device [39]. Consider the exponential cost

[TABLE]

for some $\alpha,\beta>0$. Moreover, for illustration purposes, we assume that the fusion cost function is D\mathopen{}\mathclose{{}\left(N}\right)=\gamma\mathopen{}\mathclose{{}\left(N-1}\right), for $N\geq 1$ and $\gamma>0$. Then the total cost function is given by

[TABLE]

for any positive integer $N$. In Fig. 2, we plot this total cost function with parameters $\alpha=1,\beta=1,\gamma=1,C_{\mathrm{min}}=7$ for different values of the target MSE $\tau>0$. We observe that Fig. 2 illustrates how N_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\tau}\right) increases as $\tau$ decreases, as discussed in this section. In particular, we note that N_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\tau}\right)=1,6,13 for $\tau=2,0.1,0.05$, respectively.

Finally, the total cost function (15) yields

[TABLE]

implying V\mathopen{}\mathclose{{}\left(\tau}\right)\rightarrow\infty as $\tau\rightarrow 0$. Hence there exists a threshold

[TABLE]

such that a_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\tau}\right)=1 when $\tau\geq T$, and a_{\mathrm{o}}\mathopen{}\mathclose{{}\left(\tau}\right)>1 when $\tau<T$. These cases are illustrated in Fig. 3 for $C_{\mathrm{min}}=7,\gamma=1$.

5.3 Crowdsourcing

Crowdsourcing assigns a task to a large number of less expensive but unreliable workers, instead of a small number of more expensive and reliable experts. Monetary payment to incentivize workers has been shown to affect the quality and the quantity of work in such scenarios [49]. Recently, motivated by reliability issues of crowdsourced workers and limited budgets, several researchers have pursued the limits of achievable performance from estimation-theoretic [49], information-theoretic [50], optimization [51, 52], and empirical [53] perspectives.

The authors of [53] studied the relation between monetary incentives and work quality in a knowledge task. More precisely, they performed an experiment on 451 unique workers on Amazon Mechanical Turk, and investigated the effect of bonus payments on the work quality in the task of proofreading an article. They measured the quality by the number of typographical errors found in a given article. In this scenario, each worker is paid a base salary (minimum cost), and an additional bonus (incremental cost), which is shown to yield an improvement in the work quality. In this sense, the bonus payment, i.e., the incremental cost, can be viewed as a function of the number of errors found. In particular, experiments in [53] showed that increasing the bonus payment has diminishing returns in terms of the work quality. That is, the incremental cost is a convex function of the work quality.

More recently, Lahouti and Hassibi [50] considered the crowdsourcing problem as a human-based computation problem where the main task is inference. They formulated an information-theoretic framework, where unreliable workers are modeled as parallel noisy communication channels. They represented the queries of the workers and the final inference using a joint source channel encoding/decoding scheme. Similarly, Khetan and Oh [52] studied the tradeoff between budget and accuracy in crowdsourcing scenarios under the generalized Dawid-Skene model, where they introduced an adaptive scheme to allocate a budget across unreliable workers.

We observe that there is a tradeoff between cost (monetary payments, bonus) and fidelity (quality of work) in a wide range of crowdsourcing scenarios. In particular, assigning a task to several workers, distributing the limited budget among them, and fusing their unreliable outputs have been problems of interest in the crowdsourcing literature. In this sense, our cost-fidelity formulation and repetition-based approaches may have relevance in crowdsourcing problems.

6 Conclusion and Future Directions

We considered fusing outcomes of several unreliable computational units that perform the same task. We modeled unreliability in a computational outcome using an additive perturbation, where the fidelity is inversely related to the variance of the perturbation. We investigated cost-performance tradeoffs achievable through repetition-based approaches. Here, each computational unit incurs a baseline cost as well as an incremental cost, which is a function of its fidelity.

We defined a class of repetition-based strategies, where any repetition-based strategy distributes the cost across several unreliable computational units and fuses their outcomes to produce a final output, where it incurs cost to perform the fusion operation. We considered the MSE of each strategy in estimating the error-free computation. In particular, we defined the optimal repetition-based strategy as the one incurring the smallest cost while achieving the desired MSE performance.

When the cost is a convex function of fidelity, the optimal repetition-based strategy may distribute cost across several less reliable computational units instead of using a single more reliable unit under certain conditions. For the classes of concave and linear cost functions we preserved that the optimal strategy uses only a single and relatively reliable computational unit, instead of a fusion of several less costly but less reliable units.

We assumed that outcomes produced by different computational units are uncorrelated. This framework can be extended to a correlated outcome setting. When studying the fundamental tradeoff between cost and performance, we assumed that the fusion operation is error-free. We can extend this to the case where the fusion operation also produces noisy results under cost and fidelity constraints. Moreover, we focused on a particular fusion operation, i.e., linear combination, which is common in certain applications. More generally, we can consider nonlinear fusion rules to compute the final estimate of the error-free computation. For instance, midrange [49] and median-of-means [54] estimators have been considered as alternatives to linear estimators under different scenarios to improve performance. Extension of this setup would be of interest for different network topologies, as opposed to the centralized fusion setting of this paper, as in [55].

Acknowledgements

We thank Dmitri B. Chklovskii for providing data from [19].

Appendix A Proof of Lemma 1

The MSE of the strategy $S_{N}$ with a given \boldsymbol{\theta}\in\mathopen{}\mathclose{{}\left(0,\infty}\right)^{N} is

[TABLE]

where (2) is substituted in (3). Since $Y$ and $\mathbf{U}_{\scriptstyle\boldsymbol{\theta}}$ are uncorrelated:

[TABLE]

where $\Sigma_{\mathbf{U}_{\scriptstyle\boldsymbol{\theta}}}$ is the covariance matrix of the perturbation vector $\mathbf{U}_{\scriptstyle\boldsymbol{\theta}}$. If we impose the condition that $\mathbf{w}^{T}\mathbf{1}=1$ in (A.1), then

[TABLE]

To minimize this over weights that satisfy $\mathbf{w}^{T}\mathbf{1}=1$, we first form the Lagrangian

[TABLE]

and then compute the gradient with respect to $\mathbf{w}$ to get

[TABLE]

which is satisfied iff $\mathbf{w}=\lambda\Sigma_{\mathbf{U}_{\scriptstyle\boldsymbol{\theta}}}^{-1}\mathbf{1}$. With $\mathbf{w}^{T}\mathbf{1}=1$, it yields

[TABLE]

which yields the optimal weights

[TABLE]

Finally, when substituted in \mathrm{MSE}\mathopen{}\mathclose{{}\left(\mathbf{w},\boldsymbol{\theta}}\right), we achieve

[TABLE]

The proof follows by noting \Sigma_{\mathbf{U}_{\scriptstyle\boldsymbol{\theta}}}=\mathrm{diag}\mathopen{}\mathclose{{}\left(\theta_{1}^{-1},\ldots,\theta_{N}^{-1}}\right).

Appendix B Proof of Lemma 2

We solve this optimization problem using the method of Lagrange multipliers, where we first form the Lagrangian

[TABLE]

Then, we set the derivative of the Lagrangian with respect to $\theta_{j}$ to [math], which is given by

[TABLE]

for each $j=1,\ldots,N$. Hence the necessary conditions for optimality are given by \lambda=C^{\prime}\mathopen{}\mathclose{{}\left(\theta_{j}}\right) for $j=1,\ldots,N$.

Here, we note that the cost function C\mathopen{}\mathclose{{}\left(\theta}\right) is convex and strictly increasing in $\theta$, and its derivative C^{\prime}\mathopen{}\mathclose{{}\left(\theta}\right) is nondecreasing. This implies that it is invertible, so we can write

[TABLE]

for each $j=1,\ldots,N$, where \mathopen{}\mathclose{{}\left(C^{\prime}}\right)^{-1} is the inverse of the function $C^{\prime}$. That is, $\theta_{1}=\cdots=\theta_{N}$. Moreover, by imposing the MSE constraint, we get \theta_{j}=\mathopen{}\mathclose{{}\left(\tau N}\right)^{-1} for any $j=1,\ldots,N$, which yields the desired result.

Appendix C Proof of Lemma 3

We first differentiate the total cost function as

[TABLE]

We next find its second derivative as

[TABLE]

which is nonnegative since the incremental cost function G\mathopen{}\mathclose{{}\left(\cdot}\right) and the fusion cost function D\mathopen{}\mathclose{{}\left(\cdot}\right) are both convex and $a>0$.

Appendix D Proof of Lemma 4

We first observe that from (10)

[TABLE]

for any $\tau>0$, as G\mathopen{}\mathclose{{}\left(\cdot}\right) is convex and twice differentible. Thus, the function V\mathopen{}\mathclose{{}\left(\tau}\right) is decreasing on \mathopen{}\mathclose{{}\left(0,\infty}\right). We next note that

[TABLE]

since G\mathopen{}\mathclose{{}\left(0}\right)=0 and G^{\prime}\mathopen{}\mathclose{{}\left(0}\right) is finite. Therefore, V\mathopen{}\mathclose{{}\left(\tau}\right) is nonnegative on \mathopen{}\mathclose{{}\left(0,\infty}\right). This implies that the function V\mathopen{}\mathclose{{}\left(\tau}\right) either converges to a finite limit (if and only if V\mathopen{}\mathclose{{}\left(\tau}\right) is bounded on \mathopen{}\mathclose{{}\left(0,\infty}\right)), or is unbounded as $\tau\rightarrow 0$.

Appendix E Proof of Lemma 5

Suppose that $\lambda\in[0,1]$. Since $f$ is concave, we have

[TABLE]

Then, for any $x,y>0$, we can write

[TABLE]

where we use $x/(x+y),y/(x+y)\in[0,1]$.