Bounded rational decision-making from elementary computations that reduce uncertainty

TL;DR

This paper models decision-making as elementary computations that reduce uncertainty within resource constraints, introducing a framework based on majorization and entropy to analyze bounded rationality.

Contribution

It introduces elementary computations based on probability transfers that reduce uncertainty, linking majorization theory with resource-limited decision processes.

Findings

Develops a formal framework for elementary uncertainty-reducing computations.

Proves new results on majorization, entropy, and divergence measures.

Defines resource cost functions that are monotonic with uncertainty reduction.

Abstract

In its most basic form, decision-making can be viewed as a computational process that progressively eliminates alternatives, thereby reducing uncertainty. Such processes are generally costly, meaning that the amount of uncertainty that can be reduced is limited by the amount of available computational resources. Here, we introduce the notion of elementary computation based on a fundamental principle for probability transfers that reduce uncertainty. Elementary computations can be considered as the inverse of Pigou-Dalton transfers applied to probability distributions, closely related to the concepts of majorization, T-transforms, and generalized entropies that induce a preorder on the space of probability distributions. As a consequence we can define resource cost functions that are order-preserving and therefore monotonic with respect to the uncertainty reduction. This leads to a comprehensive notion of decision-making processes with limited resources. Along the way, we prove several new results on majorization theory, as well as on entropy and divergence measures.

Figures

Equations

$$\Omega ⟶\cdots ⟶A^{\prime }⟶\varphi (A^{\prime })⟶\cdots ⟶A$$

$$\Omega \supset A^{\prime }\supset \varphi (A^{\prime })\supset A,0=C(\Omega )<C(A^{\prime })<C(\varphi (A^{\prime }))<C(A),$$

$$T_{\epsilon }:p\mapsto (p_1,\dots ,p_m+\epsilon ,\dots ,p_n-\epsilon ,\dots ,p_N),$$

$$p^{\prime }\prec p,$$

$$(Tp{)}_m=(1-\lambda )p_m+\lambda p_n,(Tp{)}_n=\lambda p_m+(1-\lambda )p_n.$$

$$A_{ij}\ge 0,Ae=e,eA=e.$$

$$\big{(}\tfrac{1}{N},\dots,\tfrac{1}{N}\big{)}\prec\big{(}\tfrac{1}{N-1},\dots,\tfrac{1}{N-1},0\big{)}\prec\big{(}\tfrac{1}{2},\tfrac{1}{2},0,\dots,0\big{)}\prec\big{(}1,0\dots,0\big{)}\,.$$

$$p=\big{(}\tfrac{1}{2},\tfrac{1}{4},\tfrac{1}{4}\big{)}\,,\quad p^{\prime}=\big{(}\tfrac{2}{5},\tfrac{2}{5},\tfrac{1}{5}\big{)}$$

$$p^{\prime }\prec p\Rightarrow C(p^{\prime })\le C(p),$$

$$C(p)=\sum _{i=1}^{N}f(p_i),$$

$$C(Z)\ge C(X)+C(Y|X),$$

$$C(p_1,\dots ,p_N)\ge C(p_1+p_2,p_3,\dots ,p_N)+(p_1+p_2)C(\frac{p_1}{p_1+p_2},\frac{p_2}{p_1+p_2}).$$

$$p⋨\varphi (p),C(p)<C(\varphi (p))\forall p\in {\mathbb{P}}_{\Omega }.$$

$$\big{(}\tfrac{1}{N},\dots,\tfrac{1}{N}\big{)}\,\prec\,p\qquad\forall p\in\mathbb{P}_{\Omega}\,.$$

$$|A_i|=\alpha q_i\forall i\in \{1,\dots ,N\},$$

$$Q(A_i)=\sum _{j=1}^{|A_i|}\frac{1}{\alpha }=q_i\forall i\in \{1,\dots ,N\},$$

$$P(\omega )=\frac{1}{\alpha }\frac{p_i}{q_i}\forall \omega \in A_i.$$

$$p^{\prime }{\prec }_{q}p:\iff {\Lambda }_q{p}^{\prime }\prec {\Lambda }_{q}p$$

$$q{\prec }_{q}p\forall p\in {\mathbb{P}}_{\Omega }.$$

$$p^{\prime }{\prec }_{q}p\Rightarrow C_q(p^{\prime })\le C_q(p),$$

$$C_{q}(p)\coloneqq\sum_{i=1}^{N}q_{i}\,f\Big{(}\frac{p_{i}}{q_{i}}\Big{)},$$

$$C_q(Z)\ge C_q(X)+C_q(Y|X)$$

$$\begin{array}[]{rl}f(M^{\prime},N)\leq f(M,N)&\ \forall M\leq M^{\prime}\leq N\,,\\[5.0pt] f(M,N)\geq f(M,N^{\prime})&\ \forall M\leq N^{\prime}\leq N\,.\end{array}$$

$$C_q(p)\ge C_{q^2}(1,0)+C_{q^{N/2}}(1,0,\dots ,0)\ge \dots \ge \log N=D_{KL}(p\Vert q),$$

$$({\tilde{T}}_{\epsilon }P)(\omega )\le ({\tilde{T}}_{\epsilon }P)({\omega }^{\prime })\iff \frac{1}{\alpha }\frac{p_m}{q_m}+\frac{\epsilon }{|A_m|}\le \frac{1}{\alpha }\frac{p_n}{q_n}-\frac{\epsilon }{|A_n|}$$

$$\epsilon \le \frac{\frac{p_n}{q_n}-\frac{p_m}{q_m}}{\frac{1}{q_m}+\frac{1}{q_n}}.$$

$$T_{\epsilon }:p\mapsto (p_1,\dots ,p_m+\epsilon ,\dots ,p_n-\epsilon ,\dots ,p_N),$$

$$q⟶\cdots ⟶p^{\prime }⟶\varphi (p^{\prime })⟶\cdots ⟶p$$

$$q{⋨}_q{p}^{\prime }{⋨}_q\varphi (p^{\prime }),0=C_q(q)<C_q(p^{\prime })<C_q(\varphi (p^{\prime })),$$

$$p^{\ast}=\mathop{\mathrm{argmax}}_{p\in\Gamma}\Big{\{}\mathbb{E}_{p}[U]\,\Big{|}\,C_{q}(p)\leq C_{0}\Big{\}}\,,$$

Full text

This is a pre-print of an article published in Entropy 2019, 21(4), 375. The final authenticated version is available online at: https://www.mdpi.com/1099-4300/21/4/375

1. Introduction

In rational decision theory, uncertainty may have multiple sources that ultimately share the commonality that they reflect a lack of knowledge on the part of the decision-maker about the environment. A paramount example is the perfectly rational decision-maker [99] that has a probabilistic model of the environment and chooses its actions so as to maximize the expected utility entailed by the different choices. When we consider bounded rational decision-makers [87], we may add another source of uncertainty arising from the decision-maker’s limited processing capabilities, since the decision-maker will not only accept a single best choice, but will accept any satisficing option. Today, bounded rationality is an active research topic that crosses multiple scientific fields such as economics, political science, decision theory, game theory, computer science, and neuroscience [81, 66, 59, 12, 31, 63, 45, 89, 101, 42, 91, 92, 90, 46, 16, 70, 58, 1, 30], where uncertainty is one of the most important common denominators.

Uncertainty is often equated with Shannon entropy in information theory [85], measuring the average number of yes/no-questions that have to be answered to resolve the uncertainty. Even though Shannon entropy has many desirable properties, there are plenty of alternative suggestions for entropy measures in the literature, known as generalized entropies, such as Rényi entropy [77] or Tsallis entropy [95]. Closely related to entropies are divergence measures, which express how probability distributions differ from a given reference distribution. If the reference distribution is uniform then divergence measures can be expressed in terms of entropy measures, which is why divergences can be viewed as generalizations of entropy, for example the Kullback-Leibler divergence [52] generalizing Shannon entropy.

Here, we introduce the concept of elementary computation based on a slightly stronger notion of uncertainty than is expressed by Shannon entropy, or any other generalized entropy alone, but is equivalent to all of them combined. Equating decision-making with uncertainty reduction, this leads to a new comprehensive view of decision-making with limited resources. Our main contributions can be summarized as follows:

1. $(i)$

Based on a fundamental concept of probability transfers related to the Pigou-Dalton principle of welfare economics [23], we promote a generalized notion of uncertainty reduction of a probability distribution that we call elementary computation. This leads to a natural definition of cost functions that quantify the resource costs for uncertainty reduction necessary for decision-making. We generalize these concepts to arbitrary reference distributions. In particular, we define Pigou-Dalton-type transfers for probability distributions relative to a reference or prior distribution, which induce a preorder that is slightly stronger than Kullback-Leibler divergence, but is equivalent to the notion of divergence given by all $f$-divergences combined. We prove several new characterizations of the underlying concept, known as relative majorization. 2. $(ii)$

An interesting property of cost functions is their behavior under coarse-graining, which plays an important role in decision-making and formalizes the general notion of making abstractions. More precisely, if a decision in a set $\Omega$ is split up into two steps by partitioning $\Omega=\bigcup_{i}A_{i}$ and first deciding in the set of (coarse-grained) partitions $\{A_{i}\}_{i}$ and secondly choosing a fine-grained option inside the selected partition $A_{i}$, then it is an important question how the cost for the total decision-making process differs from the sum of the costs in each step. We show that $f$-divergences are superadditive with respect to coarse-graining, which means that decision-making costs can potentially be reduced by splitting up the decision into multiple steps. In this regard, we find evidence that the well-known property of Kullback-Leibler divergence of being additive under coarse-graining might be viewed as describing the minimal amount of processing cost that cannot be reduced by a more intelligent decision-making strategy. 3. $(iii)$

We define bounded rational decision-makers as decision-making processes that are optimizing a given utility function under a constraint on the cost function, or minimizing the cost function under a minimal requirement on expected utility. As a special case for Shannon-type information costs, we arrive at information-theoretic bounded rationality, which may form a normative baseline for bounded-optimal decision-making in the absence of process-dependent constraints. We show that bounded-optimal posteriors with informational costs trace a path through probability space that can itself be seen as an anytime decision-making process, where each step optimally trades off utility and processing costs. 4. $(iv)$

We show that Bayesian inference can be seen as a decision-making process with limited resources given by the number of available datapoints.

Section 2 deals with $(i)$ and $(ii)$, aiming at a general characterization of decision-making in terms of uncertainty reduction. Item $(iii)$ is covered in Section 3, deriving information-theoretic bounded rationality as a special case. Section 4 illustrates the concepts with an example including item $(iv)$. Sections 5 and 6 contain a general discussion and concluding remarks.

Notation

Let $\mathbb{R}$ denote the real numbers, $\mathbb{R}_{+}\coloneqq[0,\infty)$ the set of non-negative real numbers, and $\mathbb{Q}$ the rational numbers. We write $|A|$ for the number of elements contained in a countable set $A$, and $B\setminus A$ for the set difference, that is the set of elements in $B$ that are not in $A$. $\mathbb{P}_{\Omega}$ denotes the set of probability distributions on a set $\Omega$, in particular, any $p\in\mathbb{P}_{\Omega}$ is normalized so that $p(\Omega)=\mathbb{E}_{p}[1]=1$. Random variables are denoted by capital letters $X,Y,Z$, while their explicit values are denoted by small letters $x,y,z$. For the probability distribution of a random variable $X$ we write $p(X)$, and $p(x)$ for the values of $p(X)$. Correspondingly, the expectation $\mathbb{E}[f(X)]$ is also written as $\mathbb{E}_{p(X)}[f(X)]$, $\mathbb{E}_{p(X)}[f]$, or $\mathbb{E}_{p}[f]$. We also write $\langle f\rangle_{p}:=\frac{1}{n}\sum_{i=1}^{n}f(x_{n})$, to denote the approximation of $\mathbb{E}_{p}[f]$ by an average over samples $\{x_{1},\dots,x_{n}\}$ from $p\in\mathbb{P}_{\Omega}$.

2. Decision-making with limited resources

In this section, we develop the notion of a decision-making process with limited resources following the simple assumption that any decision-making process

• $(i)$

reduces uncertainty

• $(ii)$

by spending resources.

Starting from an intuitive interpretation of uncertainty and resource costs, these concepts are refined incrementally until a precise definition of a decision-making process is given at the end of this section (Definition 2.11) in terms of elementary computations. Here, a decision-making process is a comprehensive term that describes all kinds of biological as well as artificial systems that are searching for solutions to given problems, for example a human decision-maker that burns calories while thinking, or a computer that uses electric energy to run an algorithm. However, resources do not necessarily refer to a real consumable quantity but can also represent more explicit resources (like time) as a proxy, for example the number of binary comparisons in a search algorithm, the number of forward simulations in a reinforcement learning algorithm, the number of samples in a Monte Carlo algorithm, or, even more abstractly, they can express the limited availability of some source of information, like for example the number of datapoints that are available to an inference algorithm (see Section 4).

2.1. Uncertainty reduction by eliminating options

In its most basic form, the concept of decision-making can be formalized as the process of looking for a decision $x\in\Omega$ in a discrete set of options $\Omega=\{x_{1},\dots,x_{N}\}$. We say that a decision $x\in\Omega$ is certain, if repeated queries of the decision-maker will result in the same decision, and it is uncertain, if repeated queries can result in different decisions. Uncertainty reduction then corresponds to reducing the amount of uncertain options. Hence, a decision-making process that transitions from a space $\Omega$ of options to a strictly smaller subset $A\subsetneq\Omega$ reduces the amount of uncertain options from $N=|\Omega|$ to $N_{A}\coloneqq|A|<N$, with the possible goal to eventually find a single certain decision $x^{\ast}$. Such a process is generally costly, the more uncertainty is reduced the more resources it costs (Figure 1). The explicit mapping between uncertainty reduction and resource cost depends on the details of the underlying process and on which explicit quantity is taken as the resource. For example, if the resource is given by time (or any monotone function of time), then a search algorithm that eliminates options sequentially until the target value is found (linear search) is less cost efficient than an algorithm that takes a sorted list and in each step removes half of the options by comparing the mid point to the target (logarithmic search). Abstractly, any real-valued function $C$ on the power set of $\Omega$ that satisfies $C(A^{\prime})<C(A)$ whenever $A\subsetneq A^{\prime}$ might be used as a cost function in the sense that $C(A)$ quantifies the expenses of reducing the uncertainty from $\Omega$ to $A\subset\Omega$.

In utility theory decision-making is modelled as an optimization process that maximizes a so-called utility function $U:\Omega\to\mathbb{R}$ (which can itself be an expected utility with respect to a probabilistic model of the environment, in the sense of von Neumann and Morgenstern [99]). A decision-maker that is optimizing a given utility function $U$ obtains a utility of $\frac{1}{N_{A}}\sum_{x\in A}U(x)\geq\frac{1}{N}\sum_{x\in\Omega}U(x)$ on average after reducing the amount of uncertain options from $N$ to $N_{A}<N$ (see Figure 2). A decision-maker that completely reduces uncertainty by finding the optimum $x^{\ast}=\mathrm{argmax}_{x\in\Omega}U(x)$ is called rational (w.l.o.g. we can assume that $x^{\ast}$ is unique, by redefining $\Omega$ in the case when it is not). Since uncertainty reduction generally comes with a cost, a utility optimizing decision-maker with limited resources, correspondingly called bounded rational (see Section 3), in contrast will obtain only uncertain decisions from a subset $A\subset\Omega$. Such decision-makers seek satisfactory rather than optimal solutions, for example by taking the first option that satisfies a minimal utility requirement, which Herbert Simon calls a satisficing solution [87].

Summarizing, we conclude that a decision-making process with decision space $\Omega$ that successively eliminates options can be represented by a mapping $\phi$ between subsets of $\Omega$, together with a cost function $C$ that quantifies the total expenses of arriving at a given subset,

[TABLE]

such that

[TABLE]

For example, a rational decision-maker can afford $C(\{x^{\ast}\})$, whereas a decision-maker with limited resources can typically only afford uncertainty reduction with cost $C(A)<C(\{x^{\ast}\})$.

From a probabilistic perspective, a decision-making process as described above is a transition from a uniform probability distribution over $N$ options to a uniform probability distribution over $N^{\prime}<N$ options, that converges to the Dirac measure $\delta_{x^{\ast}}$ centered at $x^{\ast}$ in the fully rational limit. From this point of view, the restriction to uniform distributions is artificial. A decision-maker that is uncertain about the optimal decision $x^{\ast}$ might indeed have a bias towards a subset $A$ without completely excluding other options (the ones in $A^{c}=\Omega{\setminus}A$), so that the behavior must be properly described by a probability distribution $p\in\mathbb{P}_{\Omega}$. Therefore, in the following section, we extend (2.1) and (2.2) to transitions between probability distributions. In particular, we must replace the power set of $\Omega$ by the space of probability distributions on $\Omega$, denoted by $\mathbb{P}_{\Omega}$.

2.2. Probabilistic decision-making

Let $\Omega$ be a discrete decision space of $N\,{=}\,|\Omega|{<}\,\infty$ options, so that $\mathbb{P}_{\Omega}$ consists of discrete distributions $p$, often represented by probability vectors $p=(p_{1},\dots,p_{N})$. However, many of the concepts presented in this and the following section can be generalized to the continuous case [62, 44].

Intuitively, the uncertainty contained in a distribution $p\in\mathbb{P}_{\Omega}$ is related to the relative inequality of its entries, the more similar its entries are, the higher the uncertainty. This means that uncertainty is increased by moving some probability weight from a more likely option to a less likely option. It turns out that this simple idea leads to a concept widely known as majorization [36, 10, 72, 13, 62, 11], which has roots in the economic literature of the early 19th century [60, 74, 23], where it was introduced to describe income inequality, later known as the Pigou-Dalton Principle of Transfers. Here, the operation of moving weight from a more likely to a less likely option corresponds to the transfer of income from one individual of a population to a relatively poorer individual (also known as a Robin Hood operation [10]). Since a decision-making process can be viewed as a sequence of uncertainty reducing computations, we call the inverse of such a Pigou-Dalton transfer an elementary computation.

Definition 2.1** (Elementary computation).**

A transformation on $\mathbb{P}_{\Omega}$ of the form

[TABLE]

where $m,n$ are such that $p_{m}\leq p_{n}$, and $0<\varepsilon\leq\frac{p_{n}{-}p_{m}}{2}$, is called a Pigou-Dalton transfer (see Figure 3). We call its inverse $T^{-1}_{\varepsilon}$ an elementary computation.

Since making two probability values more similar or more dissimilar are the only two possibilities to minimally transform a probability distribution, elementary computations are the most basic principle of how uncertainty is reduced. Hence, we conclude that a distribution $p^{\prime}$ has more uncertainty than a distribution $p$ if and only if $p$ can be obtained from $p^{\prime}$ by finitely many elementary computations (and permutations, which are not considered an elementary computation due to the choice of $\varepsilon$).

Definition 2.2** (Uncertainty).**

We say that $p^{\prime}\in\mathbb{P}_{\Omega}$ contains more uncertainty than $p\in\mathbb{P}_{\Omega}$, denoted by

[TABLE]

if and only if $p$ can be obtained from $p^{\prime}$ by a finite number of elementary computations and permutations.

Note that, mathematically this defines a preorder on $\mathbb{P}_{\Omega}$, i.e. a reflexive ($p\prec p$ for all $p\in\mathbb{P}_{\Omega}$) and transitive (if $p^{\prime\prime}\prec p^{\prime}$, $p^{\prime}\prec p$ then $p^{\prime\prime}\prec p$ for all $p,p^{\prime},p^{\prime\prime}\in\mathbb{P}_{\Omega}$) binary relation.

In the literature, there are different names for the relation between $p$ and $p^{\prime}$ expressed by Definition 2.2, for example $p^{\prime}$ is called more mixed than $p$ [79], more disordered than $p$ [78], more chaotic than $p$ [13], or an average of $p$ [36]. Most commonly however, $p$ is said to majorize $p^{\prime}$, which started with the early influences of Muirhead [65], and Hardy, Littlewood, and Pólya [36] and was developed by many authors into the field of majorization theory (a standard reference is by Marshall, Olkin, and Arnold [62]), with far reaching applications until today, especially in nonequilibrium thermodynamics and quantum information theory [15, 41, 34].

There are plenty of equivalent (arguably less intuitive) characterizations of $p\prec p^{\prime}$, some of which we are summarizing below. However, one characterization makes use of a concept very closely related to Pigou-Dalton transfers, known as T-transforms [13, 62], which expresses the fact that moving some weight from a more likely option to a less likely option is equivalent to taking (weighted) averages of the two probability values. More precisely, a T-transform is a linear operator on $\mathbb{P}_{\Omega}$ with a matrix of the form $T=(1-\lambda)\mathbb{I}+\lambda\Pi$, where $\mathbb{I}$ denotes the identity matrix on $\mathbb{R}^{N}$, $\Pi$ denotes a permutation matrix of two elements, and $0\leq\lambda\leq 1$. If $\Pi$ permutes $p_{m}$ and $p_{n}$, then $(Tp)_{k}=p_{k}$ for all $k\not\in\{m,n\}$, and

[TABLE]

Hence, a T-transform considers any two probability values $p_{m}$ and $p_{n}$ of a given $p\in\mathbb{P}_{\Omega}$, calculates their weighted averages with weights $(1-\lambda,\lambda)$ and $(\lambda,1-\lambda)$, and replaces the original values with these averages. From (2.5) it follows immediately that a T-transform with parameter $0<\lambda\leq\frac{1}{2}$ and a permutation $\Pi$ of $p_{m},p_{n}$ with $p_{m}\leq p_{n}$ is a Pigou-Dalton transfer with $\varepsilon=(p_{n}-p_{m})\lambda$. Also allowing $\frac{1}{2}\leq\lambda\leq 1$ means that T-transfers include permutations, in particular, $p^{\prime}\prec p$ if and only if $p^{\prime}$ can be derived from $p$ by successive applications of finitely many T-transforms

Due to a classic result by Hardy, Littlewood and Pólya [36, p.49], this characterization can be stated in an even simpler form by using doubly stochastic matrices, i.e. matrices $A=(A_{ij})_{i,j}$ with $A_{ij}\geq 0$ and $\sum_{i}A_{ij}=1=\sum_{j}A_{ij}$ for all $i,j$. By writing $xA\coloneqq A^{T}x$ for all $x\in\mathbb{R}^{N}$, and $e\coloneqq(1,\dots,1)$, these conditions are often stated as

[TABLE]

Note that doubly stochastic matrices can be viewed as generalizations of T-transforms in the sense that a T-transform takes an average of two entries, whereas if $p^{\prime}=pA$ with a doubly stochastic matrix $A$, then $p^{\prime}_{j}=\sum_{i}A_{ij}p_{i}$ is a convex combination, or a weighted average, of $p$ with coefficients $(A_{ij})_{i}$ for each $j$. This is also, why $p^{\prime}$ is then called more mixed than $p$ [79]. Therefore, similar to T-transforms, we might expect that if $p^{\prime}$ is the result of an application of a doubly stochastic matrix, $p^{\prime}=pA$, then $p^{\prime}$ is an average of $p$ and therefore contains more uncertainty than $p$. This is exactly what is expressed by characterization $(iii)$ in the following theorem. A similar characterization of $p^{\prime}\prec p$ is that $p^{\prime}$ must be given by a convex combination of permutations of the elements of $p$ (see property $(iv)$ below).

Without having the concept of majorization, Schur proved that functions of the form $p\mapsto\sum_{i}f(p_{i})$ with a convex function $f$ are monotone with respect to the application of a doubly stochastic matrix [83] (see property $(v)$ below). Functions of this form are an important class of cost functions for probabilistic decision-makers, as we will discuss in Example Example.

Theorem 2.3** (Characterizations of $p^{\prime}\prec p$ [62]).**

For $p,p^{\prime}\in\mathbb{P}_{\Omega}$, the following are equivalent:

1. $(i)$

$p^{\prime}\prec p$*, i.e. $p^{\prime}$ contains more uncertainty than $p$ (Definition 2.2) *

** 2. $(ii)$

$p^{\prime}$* is the result of finitely many T-transforms applied to *$p$

** 3. $(iii)$

$p^{\prime}=pA$* for a doubly stochastic matrix *$A$

** 4. $(iv)$

$p^{\prime}=\sum_{k=1}^{K}\theta_{k}\Pi_{k}(p)$* where $K\in\mathbb{N}$, $\sum_{k=1}^{K}\theta_{k}=1$, $\theta_{k}\geq 0$, and $\Pi_{k}$ is a permutation for all *$k\in\{1,\dots,K\}$

** 5. $(v)$

$\sum_{i=1}^{N}f(p^{\prime}_{i})\leq\sum_{i=1}^{N}f(p_{i})$* for all continuous convex functions *$f$

** 6. $(vi)$

$\sum_{i=1}^{k}(p^{\prime}_{i})^{\downarrow}\leq\sum_{i=1}^{k}p_{i}^{\downarrow}$* for all $k\in\{1,\dots,N{-}1\}$, where $p^{\downarrow}$ denotes the decreasing rearrangement of $p$.*

As argued above, the equivalence between $(i)$ and $(ii)$ is straight-forward. The equivalences between $(ii)$, $(iii)$, and $(vi)$ are due to Muirhead [65] and Hardy, Littlewood, and Pólya [36]. The implication $(v)\Rightarrow(iii)$ is due to Karamata [47] and Hardy, Littlewood, and Pólya [37], whereas $(iii)\Rightarrow(v)$ goes back to Schur [83]. Mathematically, $(iv)$ means that $p^{\prime}$ belongs to the convex hull of all permutations of the entries of $p$, and the equivalence $(iii)\Leftrightarrow(iv)$ is known as the Birkhoff-von Neumann theorem. Here, we have stated all relations for probability vectors $p\in\mathbb{P}_{\Omega}$, even though they are usually stated for all $p,p^{\prime}\in\mathbb{R}^{N}$ with the additional requirement that $\sum_{i=1}^{N}p_{i}=\sum_{i=1}^{N}p^{\prime}_{i}$.

Condition $(vi)$ is the classical and most commonly used definition of majorization [60, 36, 62], since it is often the easiest to check in practical examples. For example, from $(vi)$ it immediately follows that uniform distributions over $N$ options contain more uncertainty than uniform distributions over $N^{\prime}<N$ options, since we have $\sum_{i=1}^{k}\frac{1}{N}=\frac{k}{N}\leqslant\frac{k}{N^{\prime}}=\sum_{i=1}^{k}\frac{1}{N^{\prime}}$ for all $k<N$, i.e. for $N\geq 3$ it follows that

[TABLE]

In particular, if $A\subset A^{\prime}\subset\Omega$, then the uniform distribution over $A$ contains less uncertainty than the uniform distribution over $A^{\prime}$, which shows that the notion of uncertainty introduced in Definition 2.2 is indeed a generalizatin of the notion of uncertainty given by the number of uncertain options introduced in the previous section.

Note that, $\prec$ only being a preorder on $\mathbb{P}_{\Omega}$, in general, two distributions $p^{\prime},p\in\mathbb{P}_{\Omega}$ are not necessarily comparable, i.e. we can have both $p^{\prime}\not\prec p$ and $p\not\prec p^{\prime}$. In Figure 4, we visualize the regions of all comparable distributions for two exemplary distributions on a three-dimensional decision space ($N=3$), represented on the two-dimensional simplex of probability vectors $p=(p_{1},p_{2},p_{3})$. For example,

[TABLE]

can not be compared under $\prec$, since $\frac{1}{2}>\frac{2}{5}$, but $\frac{3}{4}<\frac{4}{5}$.

Cost functions can now be generalized to probabilistic decision-making by noting that the property $C(A^{\prime})<C(A)$ whenever $A\subsetneq A^{\prime}$ in (2.2) means that $C$ is strictly monotonic with respect to the preorder given by set inclusion.

Definition 2.4** (Cost functions on $\mathbb{P}_{\Omega}$).**

We say that a function $C:\mathbb{P}_{\Omega}\to\mathbb{R}_{+}$ is a cost function, if it is strictly monotonically increasing with respect to the preorder $\prec$, i.e. if

[TABLE]

with equality only if $p$ and $p^{\prime}$ are equivalent, $p^{\prime}\sim p$, which is defined as $p^{\prime}\prec p$ and $p\prec p^{\prime}$. Moreover, for a parametrized family of posteriors $(p_{r})_{r\in I}$, we say that $r$ is a resource parameter with respect to a cost function $C$, if the mapping $I\mapsto\mathbb{R}_{+},r\mapsto C(p_{r})$ is strictly monotonically increasing.

Since monotonic functions with respect to majorization were first studied by Schur [83], functions with this property are usually called (strictly) Schur-convex [62, Ch. 3].

Example* (Generalized entropies).*

From $(v)$ in Theorem 2.3 it follows that functions of the form

[TABLE]

where $f$ is strictly convex, are examples of cost functions. Since many entropy measures used in the literature can be seen to be special cases of (2.9) (with a concave $f$), functions of this form are often called generalized entropies [18]. In particular, for the choice $f(t)=t\log t$, we have $C(p)=-H(p)$, where $H(p)$ denotes the Shannon entropy of $p$. Thus, if $p^{\prime}$ contains more uncertainty than $p$ in the sense of Definition 2.2 ($p^{\prime}\prec p$) then the Shannon entropy of $p^{\prime}$ is larger than the Shannon entropy of $p$ and therefore $p^{\prime}$ contains also more uncertainty in the sense of classical information theory than $p$. Similarly, for $f(t)=-\log(t)$ we obtain the (negative) Burg entropy, and for functions of the form $f(t)=\pm t^{\alpha}$ for $\alpha\in\mathbb{R}{\setminus}\{0,1\}$ we get the (negative) Tsallis entropy, where the sign is chosen depending on $\alpha$ such that $f$ is convex (see e.g. [32] for more examples). Moreover, the composition of any (strictly) monotonically increasing function $g$ with (2.9) generates another class of cost functions, which contains for example the (negative) Rényi entropy [77]. Note also that entropies of the form (2.9) are special cases of Csiszár’s f-divergences [20] for uniform reference distributions (see Example Example below). In Figure 5, several examples of cost functions are shown for $N=3$. In this case, the 2-dimensional probability simplex $\mathbb{P}_{\Omega}$ is given by the triangle in $\mathbb{R}^{3}$ with edges $(1,0,0)$, $(0,1,0)$, and $(0,0,1)$. Cost functions are visualized in terms of their level sets.

We prove in Proposition A.1 in the appendix that all cost functions of the form (2.9) are superadditive with respect to coarse-graining. This seems to be a new result and an improvement upon the fact that generalized entropies (and $f$-divergences) satisfy information monotonicity [5]. More precisely, if a decision in $\Omega$, represented by a random variable $Z$, is split up into two steps by partitioning $\Omega=\bigcup_{i\in I}A_{i}$ and first deciding about the partition $i\in I$, correspondingly described by a random variable $X$ with values in $I$, and then choosing an option inside of the selected partition $A_{i}$, represented by a random variable $Y$, i.e. $Z=(X,Y)$, then

[TABLE]

where $C(X)\coloneqq C(p(X))$ and $C(Y|X)\coloneqq\mathbb{E}_{p(X)}[C(p(Y|X))]$. For symmetric cost functions (such as (2.9)) this is equivalent to

[TABLE]

The case of equality in (2.10) and (2.11) (see Figure 6) is sometimes called separability [51], strong additivity [21], or recursivity [4], and it is often used to characterize Shannon entropy [26, 96, 48, 55, 77, 2]. In fact, we also show in the appendix (Proposition A.2) that cost functions $C$ that are additive under coarse-graining are proportional to the negative Shannon entropy $-H$. See also Example Example in the next section, where we discuss the generalization to arbitrary reference distributions.

We can now refine the notion of a decision-making process introduced in the previous section as a mapping $\phi$ together with a cost function $C$ satisfying (2.2). Instead of simply mapping from sets $A^{\prime}$ to smaller subsets $A\subsetneq A^{\prime}$ by successively eliminating options, we now allow $\phi$ to be a mapping between probability distributions such that $\phi(p)$ can be obtained from $p$ by a finite number of elementary computations (without permutations), and we require $C$ to be a cost function on $\mathbb{P}_{\Omega}$, so that

[TABLE]

Here, $C(p)$ quantifies the total costs of arriving at a distribution $p$, and $p^{\prime}\precnsim p$ means that $p^{\prime}\prec p$ and $p\not\prec p^{\prime}$. In other words, a decision-making process can be viewed as traversing probability space by moving pieces of probability from one option to another option such that uncertainty is reduced.

Up to now we have ignored one important property of a decision-making process, the distribution $q$ with minimal cost, i.e. satisfying $C(q)\leq C(p)$ for all $p$, which must be identified with the initial distribution of a decision-making process with cost function $C$. As one might expect (see Figure 5), it turns out that all cost functions according to Definition 2.4 have the same minimal element.

Proposition 2.5** (Uniform distributions are minimal).**

The uniform distribution $(\tfrac{1}{N},\dots,\tfrac{1}{N})$ is the unique minimal element in $\mathbb{P}_{\Omega}$ with respect to $\prec$, i.e.

[TABLE]

Once (2.13) is established, it follows from (2.8) that $C((\tfrac{1}{N},\dots,\tfrac{1}{N}))\leq C(p)$ for all $p$, in particular the uniform distribution corresponds to the initial state of all decision-making processes with cost function $C$ satisfying (2.12). In particular, it contains the maximum amount of uncertainty with respect to any entropy measure of the form (2.9), known as the second Khinchin axiom [51], e.g. for Shannon entropy $0\leq H(p)\leq\log N$. Proposition 2.5 follows from characterization $(iv)$ in Theorem 2.3 after noticing that every $p\in\mathbb{P}_{\Omega}$ can be transformed to a uniform distribution by permuting its elements cyclically (see Proposition A.3 in the appendix for a detailed proof).

Regarding the possibility that a decision-maker may have prior information, for example originating from the experience of previous comparable decision-making tasks, the assumption of a uniform initial distribution seems to be artificial. Therefore, in the following section we arrive at the final notion of a decision-making process by extending the results of this section to allow for arbitrary initial distributions.

2.3. Decision-making with prior knowledge

From the discussion at the end of the previous section we conclude that, in full generality, a decision-maker transitions from an initial probability distribution $q\in\mathbb{P}_{\Omega}$, called prior, to a terminal distribution $p\in\mathbb{P}_{\Omega}$, called posterior. Note that, since once eliminated options are excluded from the rest of the decision-making process, a posterior $p$ must be absolutely continuous with respect to the prior $q$, denoted by $p\ll q$, i.e. $p(x)$ can be non-zero for a given $x\in\Omega$ only if $q(x)$ is non-zero.

The notion of uncertainty (Definition 2.2) can be generalized with respect to a non-uniform prior $q\in\mathbb{P}_{\Omega}$ by viewing the probabilities $q_{i}$ as the probabilities $Q(A_{i})$ of partitions $A_{i}$ of an underlying elementary probability space $\tilde{\Omega}=\bigcup_{i}A_{i}$ of equally likely elements under $Q$, in particular $Q$ represents $q$ as the uniform distribution on $\tilde{\Omega}$ (see Figure 7). The similarity of the entries of the corresponding representation $P\in\mathbb{P}_{\tilde{\Omega}}$ of any $p\in\mathbb{P}_{\Omega}$ (its uncertainty) then contains information about how close $p$ is to $q$, which we call the relative uncertainty of $p$ with respect to $q$ (Definition 2.6 below).

The formal construction is as follows: Let $p,q\in\mathbb{P}_{\Omega}$ be such that $p\ll q$ and $q_{i}\in\mathbb{Q}$. The case when $q_{i}\in\mathbb{R}$ then follows from a simple approximation of each entry by a rational number. Let $\alpha\in\mathbb{N}$ be such that $\alpha\,q_{i}\in\mathbb{N}$ for all $i\in\{1,\dots,N\}$, for example $\alpha$ could be chosen as the least common multiple of the denominators of the $q_{i}$. The underlying elementary probability space $\tilde{\Omega}$ then consists of $\alpha$ elements and there exists a partitioning $\{A_{i}\}_{i=1,\dots,N}$ of $\tilde{\Omega}$ such that

[TABLE]

where $Q$ denotes the uniform distribution on $\tilde{\Omega}$. In particular, it follows that

[TABLE]

i.e. $Q$ represents $q$ in $\tilde{\Omega}$ with respect to the partitioning $\{A_{i}\}_{i}$. Similarly, any $p\in\mathbb{P}_{\Omega}$ can be represented as a distribution on $\tilde{\Omega}$ by requiring that $P(A_{i})=p_{i}$ for all $i\in\{1,\dots,N\}$ and letting $P$ to be constant inside of each partition, i.e. similar to (2.15) we have $P(A_{i})=|A_{i}|\,P(\omega)=p_{i}$ for all $\omega\in A_{i}$ and therefore by (2.14)

[TABLE]

Note that, if $q_{i}=0$ then $p_{i}=0$ by absolute continuity ($p\ll q$) in which case we can either exclude option $i$ from $\Omega$ or set $P(\omega)=0$.

Example*.*

For a prior $q=(\frac{1}{6},\frac{1}{2},\frac{1}{3})$ we put $\alpha=6$, so that $\tilde{\Omega}=\{\omega_{1},\dots,\omega_{6}\}$ should be partitioned as $\tilde{\Omega}=\{\omega_{1}\}\cup\{\omega_{2},\omega_{3},\omega_{4}\}\cup\{\omega_{5},\omega_{6}\}$. Then $q_{i}$ corresponds to the probability of the $i$-th partition under the uniform distribution $Q=\frac{1}{6}(1,\dots,1)$, while $p=(\frac{1}{6},\frac{3}{4},\frac{1}{12})$ is represented on $\tilde{\Omega}$ by P=\big{(}\tfrac{1}{6},\tfrac{1}{4},\tfrac{1}{4},\tfrac{1}{4},\tfrac{1}{24},\tfrac{1}{24}) (see Figure 7).

Importantly, if the components of the representation $\Lambda_{q}p\coloneqq P$ in $\mathbb{P}_{\tilde{\Omega}}$ given by (2.16) are similar to each other, i.e. if $P$ is close to uniform, then the components of $p$ must be very similar to the components of $q$, which we express by the concept of relative uncertainty.

Definition 2.6** (Uncertainty relative to $q$).**

We say that $p^{\prime}\in\mathbb{P}_{\Omega}$ contains more uncertainty with respect to a prior $q\in\mathbb{P}_{\Omega}$ than $p\in\mathbb{P}_{\Omega}$, denoted by $p^{\prime}\prec_{q}p$, if and only if $\Lambda_{q}p^{\prime}$ contains more uncertainty than $\Lambda_{q}p$, i.e.

[TABLE]

where $\Lambda_{q}:\mathbb{P}_{\Omega}\to\mathbb{P}_{\tilde{\Omega}},p\mapsto P$ is given by (2.16).

As we will show in Theorem 2.7 below, it turns out that $\prec_{q}$ coincides with a known concept called $q$-majorization [97], majorization relative to $q$ [44, 62], or mixing distance [80]. Due to the lack of a characterization by partial sums, it is usually defined as a generalization of characterization $(iii)$ in Theorem 2.3, that is $p^{\prime}$ is $q$-majorized by $p$ iff $p^{\prime}=pA$, where $A$ is a so-called $q$-stochastic matrix, which means that it is a stochastic matrix ($Ae=e$) with $qA=q$. In particular, $\prec_{q}$ does not depend on the choice of $\alpha$ in the definition of $\Lambda_{q}$. Here, we provide two new characterizations of $q$-majorization, the one given by Definition 2.6, and one using partial sums generalizing the original definition of majorization.

Theorem 2.7** (Characterizations of $p^{\prime}\prec_{q}p$).**

The following are equivalent

1. $(i)$

$p^{\prime}\prec_{q}p$, i.e. $p^{\prime}$ contains more uncertainty relative to $q$ than $p$ (Def. 2.6)

** 2. $(ii)$

$\Lambda_{q}p$* can be obtained from $\Lambda_{q}p^{\prime}$ by a finite number of elementary computations and permutations on *$\mathbb{P}_{\tilde{\Omega}}$

** 3. $(iii)$

$p^{\prime}=pA$* for a $q$-stochastic matrix $A$, i.e. $Ae=e$ and *$qA=q$

** 4. $(iv)$

\sum_{i=1}^{N}q_{i}f\big{(}\frac{p^{\prime}_{i}}{q_{i}}\big{)}\leq\sum_{i=1}^{N}q_{i}f\big{(}\frac{p_{i}}{q_{i}}\big{)}* for all continuous convex functions *$f$

** 5. $(v)$

$\sum_{i=1}^{l-1}(p^{\prime}_{i})^{\downarrow}+a_{q}(k,l)(p^{\prime}_{l})^{\downarrow}\ \leq\ \sum_{i=1}^{l-1}p_{i}^{\downarrow}+a_{q}(k,l)p_{l}^{\downarrow}$* for all $k,l\in\mathbb{N}$ that satisfy $\alpha\sum_{i=1}^{l-1}q_{i}^{\downarrow}\leq k\leq\alpha\sum_{i=1}^{l}q_{i}^{\downarrow}$ and $1\leq l\leq N$, where the arrows indicate that $(p_{i}^{\downarrow}/q_{i}^{\downarrow})_{i}$ is ordered decreasingly, and $a_{q}(k,l)\coloneqq(\frac{k}{\alpha}-\sum_{i=1}^{l-1}q_{i}^{\downarrow})/q_{l}^{\downarrow}$.*

To prove that $(i)$, $(iii)$, and $(v)$ are equivalent (see Proposition A.4 in the appendix), we use of the fact that $\Lambda_{q}:\mathbb{P}_{\Omega}\to\mathbb{P}_{\tilde{\Omega}}$ has a left inverse $\Lambda_{q}^{-1}:\Lambda_{q}(\mathbb{P}_{\Omega})\to\mathbb{P}_{\Omega}$. This can be verified by simply multiplying the corresponding matrices given in the proof of Proposition A.4. The equivalence betweeen $(iii)$ and $(iv)$ is shown in [44] (see also [80, 62]). Characterization $(ii)$ follows immediately from Definition 2.2 and Definition 2.6.

As required from the discussion at the end of the previous section, $q$ is indeed minimal with respect to $\prec_{q}$, which means that it contains the most amount of uncertainty with respect to itself.

Proposition 2.8** (The prior is minimal).**

The prior $q\in\mathbb{P}_{\Omega}$ is the unique minimal element in $\mathbb{P}_{\Omega}$ with respect to $\prec_{q}$, that is

[TABLE]

This follows more or less directly from Proposition 2.5 and the equivalence of $(i)$ and $(iii)$ in Theorem 2.7 (see Proposition A.5 in the appendix for a detailed proof).

Order-preserving functions with respect to $\prec_{q}$ generalize cost functions introduced in the previous section (Definition 2.4). According to Proposition 2.8, such functions have a unique minimum given by the prior $q$. Since cost functions are used in Definition 2.11 below to quantify the expenses of a decision-making process, we require their minimum to be zero, which can always be achieved by redefining a given cost function by an additive constant.

Definition 2.9** (Cost functions relative to $q$).**

We say that a function $C_{q}:\mathbb{P}_{\Omega}\to\mathbb{R}_{+}$ is a cost function relative to $q$, if $C_{q}(q)=0$, if it is invariant under relabeling $(q_{i},p_{i})_{i}$, and if it is strictly monotonically increasing with respect to the preorder $\prec_{q}$, that is if

[TABLE]

with equality only if $p^{\prime}\sim_{q}p$, i.e. if $p^{\prime}\prec_{q}p$ and $p\prec_{q}p^{\prime}$. Moreover, for a parametrized family of posteriors $(p_{r})_{r\in I}$, we say that $r$ is a resource parameter with respect to a cost function $C_{q}$, if the mapping $I\mapsto\mathbb{R}_{+},r\mapsto C_{q}(p_{r})$ is strictly monotonically increasing.

Similar to generalized entropy functions discussed in Example Example, in the literature there are many examples of relative cost functions, usually called divergences or measures of divergence.

Example* ($f$-divergences).*

From $(iv)$ in Theorem 2.7 it follows that functions of the form

[TABLE]

where $f$ is continuous and strictly convex with $f(1)=0$, are examples of cost functions relative to $q$. Many well-known divergence measures can be seen to belong to this class of relative cost functions, also known as Csiszár’s $f$-divergences [20]: the Kullback-Leibler divergence (or relative entropy), the squared $\ell^{2}$* distance*, the Hartley entropy, the Burg entropy, the Tsallis entropy, and many more [32, 21] (see Figure 8 for visualizations of some of them in $N=3$ relative to a non-uniform prior).

As a generalizition of Proposition A.1 (superadditivity of generalized entropies), we prove in Proposition A.6 in the appendix that $f$-divergences are superadditive under coarse-graining, that is, for $Z=(X,Y)$

[TABLE]

whenever $C_{q}(X)\coloneqq C_{q(X)}(p(X))$ and $C_{q}(Y|X)\coloneqq\mathbb{E}_{p(X)}[C_{q(Y|X)}(p(Y|X))]$. This generalizes (2.10) to the case of a non-uniform prior. Similar to entropies, the case of equality in (2.21) is sometimes called composition rule [40], chain rule [57], or recursivity [21], and is often used to characterize Kullback-Leibler divergence [40, 63, 21, 57].

Indeed, we also show in the appendix (Proposition A.7) that all additive cost functions with respect to $q$ are proportional to Kullback-Leibler divergence (relative entropy). This goes back to Hobson’s modification [40] of Shannon’s original proof [85], after establishing the following monotonicity property for uniform distributions: If $f(M,N)$ denotes the cost $C_{u_{N}}(u_{M})$ of a uniform distribution $u_{M}$ over $M$ elements relative to a uniform distribution $u_{N}$ over $N\geq M$ elements, then (see Figure 9)

[TABLE]

Note that, even though our proof of Proposition A.7 uses additivity under coarse graining to show the monotonicity property (2.22), it is easy to see that any relative cost function of the form (2.20) also satisfies (2.22) by using the convexity of $f$ in the form $f(t)\leq\tfrac{t}{s}f(s)+(1-\tfrac{t}{s})f(0)$ with $t=\frac{N^{\prime}}{M}<\frac{N}{M}=s$.

In terms of decision-making, superadditivity under coarse-graining means that decision-making costs can potentially be reduced by splitting up the decision into multiple steps, for example by a more intelligent search strategy. For example, if $N=2^{k}$ for some $k\in\mathbb{N}$ and $C_{q}$ is superadditive, then the cost for reducing uncertainty to a single option, i.e. $p=(1,0,\dots,0)$, when starting from a uniform distribution $q$, satisfies

[TABLE]

where $q^{n}\coloneqq(\frac{1}{n},\dots,\frac{1}{n})$, and we have set $C_{q^{2}}(1,0)=1$ as unit cost (corresponding to 1 bit in the case of Kullback-Leibler divergence). Thus, intuitively the property of the Kullback-Leibler divergence of being additive under coarse-graining might be viewed as describing the minimal amount of processing costs that must be contained in any cost function, because it cannot be reduced by changing the decision-making process. Therefore, in the following we call cost functions that are proportional to the Kullback-Leibler divergence simply informational costs.

In contrast to the previous section, in the definition of $\prec_{q}$ and its characterizations we have never used elementary computations on $\mathbb{P}_{\Omega}$ directly. This is due to the fact that permutations do interact with the uncertainty relative to $q$, and therefore $\prec_{q}$ cannot be characterized by a finite number of elementary computations and permutations on $\mathbb{P}_{\Omega}$. However, we can still define elementary computations relative to $q$ by the inverse of Pigou-Dalton transfers $T_{\varepsilon}$ of the form (2.3) such that $T_{\varepsilon}p\precnsim_{q}p$ for $\varepsilon>0$, which is arguably the most basic form of how to generate uncertainty with respect to $q$.

Even for small $\varepsilon$, a regular Pigou-Dalton transfer does not necessarily increase uncertainty relative to $q$, because the similarity of the components now needs to be considered with respect to $q$. Instead, we compare the components of the representation $P=\Lambda_{q}p$ of $p\in\mathbb{P}_{\Omega}$, and move some probability weight $\varepsilon\geq 0$ from $P(A_{n})$ to $P(A_{m})$ whenever $P(\omega)\leq P(\omega^{\prime})$ for $\omega\in A_{m}$ and $\omega^{\prime}\in A_{n}$, by distributing $\varepsilon$ evenly among the elements in $A_{m}$ (see Figure 10), denoted by the transformation $\tilde{T}_{\varepsilon}$. Here, $\varepsilon$ must be small enough such that the inequality $\frac{1}{\alpha}\frac{p_{m}}{q_{m}}=P(\omega)\leq P(\omega^{\prime})=\frac{1}{\alpha}\frac{p_{n}}{q_{n}}$ is invariant under $\tilde{T}_{\varepsilon}$, which means that

[TABLE]

and therefore

[TABLE]

By construction, $\tilde{T}_{\varepsilon}$ minimally increases uncertainty in $\mathbb{P}_{\tilde{\Omega}}$ while staying in the image of $\mathbb{P}_{\Omega}$ under $\Lambda_{q}$, by keeping the values of $P$ constant in each partition, and therefore $T_{\varepsilon}\coloneqq\Lambda_{q}^{-1}\tilde{T}_{\varepsilon}\Lambda_{q}$ can be considered as the most basic way of how to increase uncertainty relative to $q$.

Definition 2.10** (Elementary computation relative to $q$).**

We call a transformation on $\mathbb{P}_{\Omega}$ of the form

[TABLE]

with $m,n$ such that $\frac{p_{m}}{q_{m}}\leq\frac{p_{n}}{q_{n}}$, and $\varepsilon$ satisfying (2.23), a Pigou-Dalton transfer relative to $q$, and its inverse an elementary computation relative to $q$.

We are now in the position to state our final definition of a decision-making process.

Definition 2.11** (Decision-making process).**

A decision-making process is a gradual transformation

[TABLE]

of a prior $q\in\mathbb{P}_{\Omega}$ to a posterior $p\in\mathbb{P}_{\Omega}$, such that each step decreases uncertainty relative to $q$. This means that $p$ is obtained from $q$ by successive application of a mapping $\phi$ between probability distributions on $\Omega$, such that $\phi(p^{\prime})$ can be obtained from $p^{\prime}$ by finitely many elementary computations relative to $q$, in particular

[TABLE]

where $C_{q}(p^{\prime})$ quantifies the total costs of a distribution $p^{\prime}$, and $p^{\prime}\precnsim_{q}p$ means that $p^{\prime}\prec_{q}p$ and $p\not\prec_{q}p^{\prime}$.

In other words, a decision-making process can be viewed as traversing probability space from prior $q$ to posterior $p$ by moving pieces of probability from one option to another option such that uncertainty is reduced relative to $q$, while expending a certain amount of resources determined by the cost function $C_{q}$.

3. Bounded rationality

3.1. Bounded rational decision-making

In this section, we consider decision-making processes that trade off utility against costs. Such decision-makers either maximize a utility function subject to a constraint on the cost function, for example an author of a scientific article that optimizes the article’s quality until a deadline is reached, or minimizing the cost function subject to a utility constraint, for example a high-school student that minimizes effort such that the requirement to pass a certain class is achieved. In both cases, the decision-makers are called bounded rational, since in the limit of no resource constraints they coincide with rational decision-makers.

In general, depending on the underlying system, such an optimization process might have additional process dependent constraints that are not directly given by resource limitations, for example in cases when the optimization takes place in a parameter space that has less degrees of freedom than the full probability space $\mathbb{P}_{\Omega}$. Abstractly, this is expressed by allowing the optimization process to search only in a subset $\Gamma\subset\mathbb{P}_{\Omega}$.

Definition 3.1** (Bounded rational decision-making process).**

Let $U:\Omega\to\mathbb{R}$ be a given utility function, and $\Gamma\subset\mathbb{P}_{\Omega}$. A decision-making process with prior $q$, posterior $p^{\ast}\in\Gamma$, and cost function $C_{q}$ is called bounded rational if its posterior satisfies

[TABLE]

for a given upper bound $C_{0}\geq 0$, or equivalently

[TABLE]

for a given lower bound $U_{0}\in\mathbb{R}$. In the case when the process constraints disappear, i.e. if $\Gamma=\mathbb{P}_{\Omega}$, then a bounded rational decision-maker is called bounded-optimal.

The equivalence between (3.1) and (3.2) is easily seen from the equivalent optimization problem given by the formalism of Lagrange multipliers [25],

[TABLE]

where the cost or utility constraint is expressed by a trade-off between utility and cost, or cost and utility, with a trade-off parameter given by the Lagrange multiplier $\beta$, which is chosen such that the constraint given by $C_{0}$ or $U_{0}$ is satisfied. It is easily seen from the maximization problem on the right side of (3.3) that a larger value of $\beta$ decreases the weight of the cost term and thus allows for higher values of the cost function. Hence, $\beta$ parametrizes the amount of resources the decision-maker can afford with respect to the cost function $C_{q}$, and, at least in non-trivial cases (non-constant utilities) it is therefore a resource parameter with respect to $C_{q}$ in the sense of Definition 2.9. In particular, for $\beta\to 0$, the decision-maker minimizes its cost function irrespective of the expected utility, and therefore stays at the prior, $p_{0}=q$, whereas $\beta\to\infty$ makes the cost function disappear so that the decision-maker becomes purely rational with a Dirac posterior centered on the optima $x^{\ast}$ of the utility function $U$.

For example, in Figure 11 we can see how the posteriors $(p_{\beta})_{\beta\geq 0}$ of bounded-optimal decision-makers with different cost functions for $N=3$ and with utility $U=(0.8,1.0,0.4)$ leave a trace in probability space, by moving away from an exemplary prior $q=(\frac{1}{3},\frac{1}{2},\frac{1}{6})$ and eventually arriving at the rational solution $\delta_{(0,1,0)}$.

For informational costs (i.e. proportional to Kullback-Leibler divergence), $\beta$ is a resource parameter with respect to any cost function.

Proposition 3.2**.**

If $(p_{\beta})_{\beta\geq 0}$ is a family of bounded-optimal posteriors given by (3.3) with $C_{q}(p)=D_{KL}(p\|q)$, then $\beta$ is a resource parameter with respect to any cost function, in particular

[TABLE]

This generalizes a result in [78] to the case of non-uniform priors, by making use of our new characterization $(v)$ of $\prec_{q}$, by which it suffices to show that the function $\beta\mapsto\sum_{i=1}^{l-1}(p_{\beta,i})^{\downarrow}+a_{q}(k,l)(p_{\beta,l})^{\downarrow}$ is monotonically increasing for all $k,l$ specified in Theorem 2.7 (see Proposition A.8 in the appendix for details). For simplicity, we restrict ourselves to the case of the Kullback-Leibler divergence, however the proof is analogous for cost functions of the form (2.20) with $f$ being differentiable and strictly convex on $[0,1]$ (so that $f^{\prime}$ is strictly monotonically increasing and thus invertible on $[0,1]$, see [78] for the case of uniform priors).

Hence, for any $\beta>0$, the posteriors $(p_{\beta^{\prime}})_{\beta^{\prime}<\beta}$ of a bounded-optimal decision-making process with the Kullback-Leibler divergence as cost function can be regarded as the steps of a decision-making process (i.e. satisfying (2.25)) with posterior $p_{\beta}$, where each step optimally trades off utility against informational cost. This means that with increasing $\beta$ the posteriors $p_{\beta}$ do not only decrease entropy in the sense of the Kullback-Leibler divergence, but also in the sense of any other cost function.

The important case of bounded-optimal decision-makers with informational costs is termed information-theoretic bounded rationality [68, 92, 70] and is studied more closely in the following sections.

3.2. Information-theoretic bounded rationality

For bounded-optimal decision-making processes with informational costs the unconstrained optimization problem (3.3) takes the form $\max_{p\in\mathbb{P}_{\Omega}}\mathcal{F}[p]$, where

[TABLE]

which has a unique maximum $p_{\beta}$, the bounded-optimal posterior given by

[TABLE]

with normalization constant $Z_{\beta}$. This form can easily be derived by finding the zeros of the functional derivative of the objective functional (3.5) with respect to $p$ (with an additional normalization constraint), whereas the uniqueness follows from the convexity of the mapping $p\mapsto D_{KL}(p\|q)$. For the actual maximum of $\mathcal{F}$ we obtain

[TABLE]

so that $p_{\beta}(x)=q(x)\,e^{\beta(U(x)-\mathcal{F}_{\beta})}$.

Due to its analogy with physics, in particular thermodynamics (see e.g. [70]), the maximization of (3.5) is known as the Free Energy principle of information-theoretic bounded rationality, pioneered in [68, 92, 70], further developed in [29, 33], and applied in recent studies of artificial systems, such as generative neural networks trained by Markov chain Monte Carlo methods [38], or in reinforcement learning as an adaptive regularization strategy [56, 35], as well as in recent experimental studies on human behavior [71, 82]. Note that there is a formal connection of (3.5) and the Free Energy principle of active inference [28], however, as discussed in [33, Sec. 6.3], both Free Energy principles have conceptually different interpretations.

Example* (Bayes rule as a bounded-optimal posterior).*

In Bayesian inference, the parameter $\theta$ of the distribution $p_{\theta}$ of a random variable $Y$ is inferred from a given dataset $d=\{y_{1},\dots,y_{N}\}$ of observations of $Y$ by treating the parameter itself as a random variable $\Theta$ with a prior distribution $q(\Theta)$. The parametrized distribution of $Y$ evaluated at an observation $y_{i}\in d$ given a certain value of $\Theta$, i.e. $p(y_{i}|\Theta{=}\theta)$, is then understood as a function of $\theta$, known as the likelihood of the datapoint $y_{i}$ under the assumption of $\Theta=\theta$. After seeing the dataset $d$, the belief about $\Theta$ is updated by using Bayes rule

[TABLE]

This takes the form of a bounded-optimal posterior (3.6) with $\beta=N$ and utility function given by the average log-likelihood per datapoint,

[TABLE]

since then Bayes rule reads

[TABLE]

The corresponding Free Energy (3.5), that is maximized by (3.7),

[TABLE]

coincides with the variational Free Energy $\mathcal{F}_{var}$ from Bayesian statistics. Indeed, from (3.8) it is easy to see that $\mathcal{F}$ assumes its maximum when $p(\Theta)$ is proportional to $q(\Theta)p(d|\Theta)$, that is when $p(\Theta)$ is given by (3.7). In the literature, $\mathcal{F}_{var}$ is used in the variational characterization of Bayes rule, in cases when the form (3.7) cannot be achieved exactly but instead is approximated by optimizing (3.8) over the parameters $\vartheta$ of a parametrized distribution $p_{\vartheta}(\Theta)$ [39, 61].

In the following section, we will see that the Free Energy $\mathcal{F}$ of a bounded rational decision-making process satisfies a recursivity property, which allows the interpretation of $\mathcal{F}$ as a certainty-equivalent.

3.3. The recursivity of $\mathcal{F}$ and the value of a decision problem

Consider a bounded-optimal decision-maker with an informational cost function deciding about a random variable $Z$ with values in $\Omega$ that is decomposed into the random variables $X$ and $Y$, i.e. $Z=(X,Y)$. This decomposition can be understood as a two-step process, where first a decision about a partition $A_{i}$ of the full search space $\Omega=\bigcup_{i\in I}A_{i}$ is made, represented by a random variable $X$ with values in $I$, followed by a decision about $Y$ inside the partition selected by $X$ (see Figure 6).

Since $p(Z)=p(X)p(Y|X)$, by the additivity of the Kullback-Leibler divergence (Proposition A.7), we have

[TABLE]

and therefore, if $\mathcal{F}_{\beta}[p(Y|X)]\coloneqq\mathbb{E}_{p(Y|X)}[U(X,Y)]-\frac{1}{\beta}D_{KL}(p(Y|X)\|q(Y|X))$ denotes the Free Energy of the second step,

[TABLE]

In particular, the Free Energy $\mathcal{F}_{\beta}[p(Y|X)]$ of the second decision-step plays the role of the utility function of the first decision-step (see Figure 12). In Equation (3.9), the two decision-steps have the same resource parameter $\beta$, controlling the strength of the constraint on the total informational costs

[TABLE]

More generally, each step might have a separate information-processing constraint, which requires two resource parameters $\beta_{1}$ and $\beta_{2}$, and results in the total Free Energy

[TABLE]

Example*.*

Consider a bounded-rational decision-maker with informational cost function and a utility function $U$ defined on a set $\Omega=\{z_{1},\dots,z_{4}\}$ with values as given in Figure 13 and an information-processing bound of $0.2$ bits ($\beta\approx 0.9$). If we partition $\Omega$ into the disjoint subsets $\{z_{1},z_{2}\}$ and $\{z_{3},z_{4}\}$, then the decision about $Z$ can be decomposed into two steps, $Z=(X,Y)$, the decision about $X$ corresponding to the choice of the partition and the decision about $Y$ given $X$ corresponding to the choice of $z_{i}$ inside the given partition determined by $X$. According to Equation (3.9), the choice of the partition $X=x_{i}$ is not in favor of the achieved expected utility inside each partition, but of the Free Energy (see Figure 13).

Therefore, a bounded rational decision-maker that has the choice among decision-problems ideally should base its decision not on the expected utility that might be achieved but on the Free Energy of the subordinate problems. In other words, the Free Energy quantifies the value of a decision-problem, that besides of the achieved average utility also takes the information-processing costs into account.

3.4. Multi-task decision-making and the optimal prior

So far, we have considered decision-making problems with utility functions defined on $\Omega$ only, modelling a single decision-making task. This is extended to multi-task decision-making problems by utility functions of the form $U:\mathcal{W}\times\Omega\to\mathbb{R},(w,x)\mapsto U(w,x)$, where the additional variable $w\in\mathcal{W}$ represents the current state of the world. Different world states $w$ in general lead to different optimal decisions $x^{\ast}(w)\coloneqq\mathrm{argmax}_{x\in\Omega}U(w,x)$. For example, in a chess game the optimal moves depend on the current board configurations the players are faced with.

The prior $q$ for a bounded-rational multi-task decision-making problem may either depend or not depend on the world state $w\in\mathcal{W}$. In the first case, the multi-task decision-making problem is just given by multiple single-task problems, i.e. for each $w\in\mathcal{W}$, $q(X|W=w)$ and $p(X|W=w)$ are the prior and posterior of a bounded rational decision-making process with utility function $x\mapsto U(x,w)$, as described in the previous sections. In the case when there is a single prior for all world states $w\in\mathcal{W}$, the Free Energy is

[TABLE]

where $p(W)$ is a given world state distribution. Note that, for simplicity we assume that $\beta$ is independent of $w\in\mathcal{W}$, which means that only the average information-processing is constrained, in contrast to the information-processing being constrained for each world state which in general would result in $\beta$ being a function of $w$. Similarly to single-task decision-making (Equation (3.6)) the maximum of (3.10) is achieved by

[TABLE]

with normalization constant $Z_{\beta}(w)$. Interestingly, the deliberation cost in (3.10) depends on how well the prior was chosen to reach all posteriors without violating the processing constraint. In fact, viewing the Free Energy (3.10) as a function of both, posterior and prior, $\mathcal{F}[p(X|W)]=\mathcal{F}[p(X|W),q(X)]$, and optimizing for the prior yields the marginal of the joint distribution $p(W,X)=p(W)p(X|W)$, i.e. the mean of the posteriors for the different world states,

[TABLE]

Similarly to (3.6), Equation (3.12) follows from finding the zeros of the functional derivative of the Free Energy with respect to $q(X)$ (modified by an additional term for the normalization constraint).

Optimizing the Free Energy $\mathcal{F}[p(X|W),q(X)]$ for both prior and posterior can be achieved by iterating Equations (3.11) and (3.12). This results in an alternating optimization algorithm, originally developed independently by Blahut and Arimoto to calculate the capacity of a memoryless channel [14, 9] (see [22] for a convergence proof by Csiszár and Tusnády). Note that

[TABLE]

in particular, that the information-processing cost is now given by the mutual information $I(W;X)$ between the random variables $W$ and $X$. In this form, we can see that the Free Energy optimization with respect to prior and posterior is equivalent to the optimization problem in classical rate distortion theory [86], where $U$ is given by the negative of the distortion measure.

Similarly as in rate-distortion theory, where compression algorithms are anaylized with respect to the rate-distortion function, any decision-making system can now be analyzed with respect to informational bounded-optimality. More precisely, when plotting the achieved expected utility against the information-processing resources of a bounded-rational decision-maker with optimal prior, we obtain a pareto-optimality curve that forms an efficiency-frontier that cannot be surpassed by any decision-making process (see Figure 14$(c)$).

3.5. Multi-task decision-making with unknown world state distribution

A bounded rational decision-making process with informational cost and utility $U:\mathcal{W}\times\Omega\to\mathbb{R}$ that has an optimal prior $q^{\ast}(X)$ given by the marginal (3.12) must have perfect knowledge about the world state distribution $p(W)$. In contrast, here we consider the case when the exact shape of the world state distribution is unknown to the decision-maker and therefore has to be inferred from the already seen world states. More precisely, we assume that the world state distribution is parametrized by a parameter $\theta\in\mathbb{R}$, i.e. $p(W)=p_{\theta_{true}}(W)$ for a given parametrized distribution $p_{\theta}(W)$. Since the true parameter $\theta_{true}$ is unknown, $\theta$ is treated as a random variable by itself, so that $p_{\theta}(W)=p(W|\Theta=\theta)$. After a dataset $d=\{w_{1},\dots,w_{N}\}\in\mathcal{W}^{N}$ of samples from $p(W|\Theta=\theta_{true})$ has been observed the joint distribution of all involved random variables can be written as

[TABLE]

where $p(\Theta)$ denotes the decision-maker’s prior belief about $\Theta$, and $p(D=d|\Theta)=\prod_{i=1}^{N}p(w_{i}|\Theta)$ is the likelihood of the previously observed world states. Therefore, the resulting (multi-task) Free Energy (see Equation (3.10)) is given by

[TABLE]

It turns out that we obtain Bayesian inference as a byproduct of optimizing (3.13) with respect to the prior $q(X|D)$. Indeed, by calculating the functional derivative with respect to $q(X|D)$ of the Free Energy (3.13) plus an additional term for the normalization constraint of $q(X|D)$ (with Lagrange multiplier $\lambda$), we can see that any distribution $q^{\ast}(X|D)$ that optimizes (3.13) must satisfy

[TABLE]

where $\lambda\in\mathbb{R}$ is chosen such that $q^{\ast}(X|D{=}d)\in\mathbb{P}_{\Omega}$ for any $d\in\mathcal{W}^{N}$. This is equivalent to

[TABLE]

where $Z_{D}$ denotes the normalization constant of $q^{\ast}(X|D)$, given by $Z_{D}=\mathbb{E}_{p(\Theta)}[p(D|\Theta)]$, since $\mathbb{E}_{p(X|D,W)}[1]=1$ as well as $\mathbb{E}_{p(W|\Theta)}[1]=1$. Therefore, we obtain

[TABLE]

with $p^{\ast}(\Theta|D)$ as defined in Equation (3.14). Hence, we have shown

Proposition 3.3** (Optimality of Bayesian inference).**

The optimal prior $q^{\ast}(X|D)$ that maximizes (3.13) is given by $q^{\ast}(X|D)=\mathbb{E}_{p^{\ast}(\Theta|D)p(W|\Theta)}[p(X|D,W)]$, where $p^{\ast}(\Theta|D)$ is the Bayes posterior

[TABLE]

4. Example: Absolute identification task with known and unknown stimulus distribution

Consider a bounded rational decision-maker with a multi-task utility function $U$ such that for each $w\in\mathcal{W}$, $U(w,x)$ is non-zero for only one choice $x\in\Omega$ as shown in Figure 14. Here, the decision and world spaces are both finite sets of $N=20$ elements. The world state distribution $p(W)$ is given by a mixture of two Gaussian distributions as shown in Figure 14$(b)$. Due to some world states $w\in\mathcal{W}$ being more likely than others, there are some choices $x\in\Omega$ that are less likely to be optimal.

4.1. Known stimulus distribution

As can be seen in Figure 14$(c)$ (dashed line), here it is not ideal to have a uniform prior distribution, $q(x)=\frac{1}{N}$ for all $x\in\Omega$. Instead, if the world state distribution is known perfectly and the prior has the form suggested by Equation (3.12), i.e. $q(x)=\sum_{w}p(w)p_{\beta}(x|w)$, then, as can be seen in Figure 14$(c)$ (solid line), achieving the same expected utility as with a uniform prior requires less informational resources. In particular, the explicit form of $q^{\ast}$ depends on the resource parameter $\beta$, see Figure 14$(d)$. For low resource availability (small $\beta$), only the choices that correspond to the most probable world states are considered. However, for $\beta\to\infty$, we have

[TABLE]

because here $\lim_{\beta\to\infty}p_{\beta}(x|w)=\delta_{w,x}$ is the posterior of a rational decision-maker, where $\delta_{w,x}$ denotes the Kronecker-$\delta$ (which is only non-zero if $w{=}x$). Hence, for decision-makers with abundant information-processing resources (large $\beta$) the optimal prior $q^{\ast}(X)$ approaches the form of the world state distribution $p(W)$ (since here $\mathcal{W}=\Omega$).

4.2. Unknown stimulus distribution

In the case when the decision-maker has to infer its knowledge about $p(W)$ from a set of samples $d=\{w_{1},\dots,w_{N}\}$, we know from Section 3.5 that this is optimally done via Bayesian inference. Here, we assume a mixture auf two Gaussians as a parametrization of $p(W)$, so that $\theta=(\mu_{1},\mu_{2},\sigma_{1},\sigma_{2})$, where $\mu_{i}$ and $\sigma_{i}$ denote mean and standard-deviation of the $i$-th component respectively (for simplicity, with fixed equal weights for the two mixture components).

In Figure 15$(a)$, we can see how different values of $N$ affect the belief about the world state distribution, $p(W|D)=\mathbb{E}_{p(\Theta|D)}[p(W|\Theta)]$, when $p(\Theta|D)$ is given by the Bayes posterior (3.14) with a uniform prior belief $p(\Theta)$. The resulting expected utilities (averaged over samples from $p(D|\theta_{true})$) as functions of available information-processing resources are displayed in Figure 15$(b)$, which shows how the performance of a bounded-rational decision-maker with optimal prior and perfect knowledge about the true world state distribution is approached by bounded rational decision-makers with limited but increasing knowledge given by the sample size $N$.

Abstractly, we can view Equation (3.14) as the bounded optimal solution to the decision-making problem that starts with a prior $p(\Theta)$ and arrives at a posterior $p(\Theta|D\,{=}\,d)$ after processing the samples in $d=\{w_{1},\dots,w_{N}\}$ (see also Example Example). In fact, the posteriors shown in Figure 15$(a)$ satisfy the requirements for a decision-making process with resource given by the number of datapoints $N$, when averaged over $p(D)$. In particular, by increasing $N$ the posteriors contain less and less uncertainty with respect to the preorder $\prec$ given by majorization. Accordingly, if we plot the achieved expected utility against the number of samples, we obtain an optimality curve similar to Figure 14$(c)$ and Figure 15$(b)$. In Figure 16 we can see how Bayesian Inference outperforms Maximum Likelihood when evaluated with respect to the average expected utility of a bounded-rational decision-maker with $2$ bits of information-processing resources.

5. Discussion

In this work, we have developed a generalized notion of decision-making in terms of uncertainty reduction. Based on the simple idea of transferring pieces of probability between the elements of a probability distribution, which we call elementary computations, we have promoted a notion of uncertainty which is known in the literature as majorization, a preorder $\prec$ on $\mathbb{P}_{\Omega}$. Taking non-uniform initial distributions into account, we extended the concept to the notion of relative uncertainty, which corresponds to relative majorization $\prec_{q}$. Despite of the large amount of research that has been done on majorization theory, from the early works [65, 60, 36] through further developments [79, 10, 8, 72, 19, 13, 62] to modern applications [15, 41, 34], there is a lack of results on the more general concept of relative majorization. This does not seem to be due to a lack of interest, as can be seen from the results [97, 80, 44, 54], but mostly because relative majorization looses some of the appealing properties of majorization which makes it harder to deal with, for example that permutations no longer leave the ordering $\prec_{q}$ invariant, in contrast to the case of a uniform prior. This restriction does, however, not affect our application of the concept to decision-making, as permutations are not considered as elementary computations, since they do not diminish uncertainty. By reducing the non-uniform to the uniform case, we managed to prove new results on relative majorization (Theorem 2.7), which then enabled new results in other parts of the paper (Example Example, Propositions A.6, A.8), and allowed an intuitive interpretation of our final definition of a decision-making process (Definition 2.11) in terms of elementary computations with respect to non-uniform priors (Definition 2.10).

More precisely, starting from stepwise elimination of uncertain options (Section 2.3), we have argued that decision-making can be formalized by transitions between probability distributions (Section 2.2), and arrived at the concept of decision-making processes traversing probability space from prior to posterior by successively moving pieces of probability between options such that uncertainty relative to the prior is reduced (Section 2.1). Such transformations can be quantified by cost functions, which we define as order-preserving functions with respect to $\prec_{q}$ and capture the resource costs that must be expended by the process. We have shown (Propositions A.1, A.6) that many known generalized entropies and divergences, which are examples of such cost functions (Examples Example and Example), satisfy superadditivity with respect to coarse-graining. This means that under such cost functions, decision-making costs can potentially be reduced by a more intelligent search strategy, in contrast to Kullback-Leibler divergence, which was characterized as the only additive cost function (Proposition A.7). There are plenty of open questions for further investigation in that regard. First, it is not clear under which assumptions on the cost functions $C_{q}$ superadditivity could be improved to $C_{q}(p)=\alpha D_{KL}(p\|q)+r(p,q)$ with $\alpha>0$ and $r(p,q)\geq 0$. Additionally, it would be an interesting challenge to find sufficient conditions implying super-additivity that include more cost functions than $f$-divergences. The field of information geometry might give further insights on the topic, since there are studies in similar directions, in particular characterizations of divergence measures in terms of information monotonicity and the data-processing inequality [43, 5, 6, 7]. One interesting result is the characterization of Kullback-Leibler divergence as the single divergence measure being both an $f$-divergence and a Bregman divergence.

In Section 3, bounded rational decision-makers were defined as decision-making processes that are maximizing utility under constraints on the cost function, or equivalently minimizing resource costs under a minimal utility requirement. In the important case of additive cost functions (i.e. proportional to Kullback-Leibler divergence), this leads to information-theoretic bounded rationality [68, 92, 70, 29, 33], which has precursors in the economic and game-theoretic literature [64, 66, 63, 101, 90, 42, 93, 91, 92, 46, 100, 58]. We have shown that the posteriors of a bounded rational decision-maker with increasing informational constraints leave a path in probability space that can itself be considered an anytime decision-making process, in each step perfectly trading off utility against processing costs (Prop. 3.2). In particular, this means that the path of a bounded rational decision-maker with informational cost decreases uncertainty with respect to all cost functions, not just Kullback-Leibler divergence. We have also studied the role of the prior in bounded rational multi-task decision-making, where we have seen that imperfect knowledge about the world state distribution leads to Bayesian inference as a byproduct, which is in line with the characterization of Bayesian inference as minimizing prediction surprise [69], but also demonstrates the wide applicability of the developed theory of decision-making with limited resources.

Finally, in Section 4, we have presented the results of a simulated bounded rational decision-maker solving an absolute identification task with and without knowledge about the world state distribution. Additionally, we have seen that Bayesian inference can be considered a decision-making process with limited resources by itself, where the resource is given by the number of available datapoints.

6. Conclusion

To our knowledge, this is the first principled approach to decision-making based on the intuitive idea of Pigou-Dalton-type probability transfers (elementary computations). Information-theoretic bounded rationality has been introduced by other axiomatic approaches before [68, 63]. For example, in [68] a precise relation between rewards and information value was derived by postulating that systems will choose those states with high probability that are desirable for them. This leads to a direct coupling of probabilities and utility, where utility and information inherit the same structure, and only differ with respect to normalization—see [17] for similar ideas. In contrast, we assume utility and probability to be independent objects a priori that only have a strict relationship in the case of bounded-optimal posteriors. The approach in [63] introduces Kullback-Leibler divergence as disutility for decision control. Based on Hobson’s characterization [40], the authors argue that cost functions should be monotonic with respect to uniform distributions (property (2.22)) and invariant under decomposition, which coincides with additivity under coarse-graining (see Examples Example, Example). Both of these assumptions are special cases of our more general treatment, where cost functions must be monotonic with respect to elementary computations and are generally not restricted to being additive.

In the literature, there are many mechanistic models of decision-making that instantiate decision-making processes with limited resources. Examples include reinforcement learning algorithms with variable depth [49, 50], Markov chain Monte Carlo (MCMC) models where only a certain number of samples can be evaluated [100, 67, 38], and evidence accumulation models that accumulate noisy evidence until either a fixed threshold is reached [53, 75, 94, 76, 84] or where thresholds are determined dynamically by explicit cost functions depending on the number of allowed evidence accumulation steps [27, 24]. Many of these concrete models may be described abstractly by resource parameterizations (Def. 2.9). More precisely, in such cases the posteriors $\{p_{r}\}_{r\in I}\subset\Gamma\subset\mathbb{P}_{\Omega}$ are generated by an explicit process with process constraints $\Gamma$ and resource parameter $r$. For example, in diffusion processes $r$ may correspond to the amount of time allowed for evidence accumulation, in Monte Carlo algorithms $r$ may reflect the number of MCMC steps, and in a reinforcement learning agent $r$ may represent the number of forward-simulations. If the resource restriction is described by a monotonic cost function $r\mapsto c_{r}$ [27, 24], then the process can be optimized by a maximization problem of the form

[TABLE]

where $M,M^{\prime}$ are non-negative constants, $\Gamma_{r}\subset\Gamma$ denotes the subset of probability distributions with resource $r$, and $C_{q}$ denotes a cost function such that $r\mapsto C_{q}(p)$ for $p\in\Gamma_{r}$ is strictly monotonically increasing. In particular, such cases can also be regarded as bounded rational decision-making problems of the form (3.1).

Bounded rationality models in the literature come in a variety of flavors. In the heuristics and biases paradigm the notion of optimization is often dismissed in its entirety [31], even though decision-makers still have to have a notion of options being better or worse, for example in order to adapt their aspiration levels in a satisficing scheme [88]. We have argued that from an abstract normative perspective we can formulate satisficing in probabilistic terms, such that one could investigate the efficiency of heuristics within this framework. Another prominent approach to bounded rationality is given by systems capable of decision-making about decision-making, i.e. meta-decision-making. Explicit decision-making processes composed of two decision steps have been studied, for example, in the reinforcement learning literature [49, 50, 73, 98], where the first step is represented by a meta decision about whether a cheap model-free or a more expensive model-based learning algorithm is used in the second step. The meta step consists of a trade-off between the estimated utility against the decision-making costs of the second decision step. In the information-theoretic framework of bounded rationality this could be seen as a natural property of multi-step decision-making and the recursivity property (3.9) from which follows that the value of a decision-making problem is given by its Free Energy, that besides of the achieved utility also takes the corresponding processing costs into account. Another prominent approach to bounded rationality is computational rationality [58] where the focus lies on finding bounded-optimal programs that solve constrained optimization problems presented by the decision-maker’s architecture and the task environment. As described above, such architectural constraints could be represented by a process dependent subset $\Gamma\subset\mathbb{P}_{\Omega}$, and in fact our resource costs could be included into such a subset $\Gamma_{r}$ as well. From this point of view, both frameworks would look for bounded-optimal solutions in that the search space is first restricted and then the best solution in the restricted search space is found. However, our search space would consist of distributions describing probabilistic input-output maps, whereas the search space of programs would be far more detailed.

The notion of decision-making presented in this work, intuitively developed from the basic concept of uncertainty reduction given by elementary computations and motivated by the simple idea of progressively eliminating options, on the one hand provides a promising theoretical playground that is open for further investigation (e.g. superadditivity of cost functions and minimality of relative entropy), potentially providing new insights into the connection between the fields of rationality theory and information theory, and on the other hand serves the purpose of a general framework to describe and analyze all kinds of decision-making processes (e.g. in terms of bounded-optimality).

Appendix A Proofs of technical results from Sections 2 and 3

Proposition A.1** (Superadditivity of generalized entropies under coarse-graining, Example Example).**

All cost functions of the form

[TABLE]

with $f$ (strictly) convex and differentiable on $[0,1]$, and $f(1)=0$, are superadditive with respect to coarse-graining, that is

[TABLE]

whenever $Z=(X,Y)$, and $C(X)\coloneqq C(p(X))$ and $C(Y|X)\coloneqq\mathbb{E}_{p(X)}[C(p(Y|X))]$.

Proof.

As we will see in the following proof, strict convexity is not needed for superadditivity, but it is required for the definition of a cost function. First, since $\sum_{i}p_{i}=1$, notice that we can always redefine the convex function $f$ in (A.1) by $f_{c}(t)\coloneqq f(t)-c(t-1)$ for an arbitrary constant $c\in\mathbb{R}$ without changing $C(p)$ for all $p\in\mathbb{P}_{\Omega}$. Hence, without loss of generality, we may assume $f^{\prime}(1)=0$, so that $t=1$ is a global minimum of $f$ (since $f(t)\geq f(1)+(t-1)\,f^{\prime}(1)=f(1)$ due to convexity). Since $C$ is symmetric, superadditivity under coarse-graining is equivalent to

[TABLE]

This simply follows by induction, since (A.2) corresponds to the partitioning $\Omega=\bigcup_{j=1}^{N-1}A_{j}$ with $A_{1}=\{x_{1},x_{2}\}$ and $A_{j}=\{x_{j+1}\}$ for all $j=2,\dots,N-1$ (see also [3, 2.3.5]). In terms of $f$, (A.2) reads

[TABLE]

By setting $u=p_{1}+p_{2}$ and $v=\tfrac{p_{1}}{p_{1}+p_{2}}$, this is equivalent to

[TABLE]

for all $u,v\in[0,1]$. Writing $F_{v}(u)\coloneqq f(uv)+f(u(1-v))-f(u)-u\,(f(v)+f(1-v))$ and noting that $F_{v}(0)\geq 0$ and $F_{v}(1)=0$, it suffices to show that $F^{\prime}_{v}(u)\leq 0$, which shows that $F_{v}$ is monotonically decreasing from $F_{v}(0)$ to $F_{v}(1)=0$ and thus $F_{v}(u)\geq 0$ for all $u\in[0,1]$. We have for all $v,u\in[0,1]$

[TABLE]

By the symmetry of $F_{v}$ under the replacement of $v$ by $1{-}v$, without loss of generality, we may assume that $v\leq\frac{1}{2}$, so that $uv\leq u(1{-}v)\leq u$. Since $f$ is convex, $f^{\prime}$ is monotonically increasing on $[0,1]$, and thus $f^{\prime}(uv)\leq f^{\prime}(u(1{-}v)\leq f^{\prime}(u)$. In particular,

[TABLE]

and thus, since $f(v)+f(1{-}v)\geq 0$, it follows that $F^{\prime}_{v}(u)\leq 0$, which completes the proof. ∎

Proposition A.2** (Characterization of Shannon entropy, Example Example).**

If a cost function $C$ is additive under coarse-graining, that is if $C(Z)=C(X)+C(Y|X)$ with the notation from Proposition A.1, then

[TABLE]

for some $\alpha\geq 0$, i.e. $C$ is proportional to the negative Shannon entropy $-H$.

Proof.

Since uniform distributions over $N$ options are majorized by uniform distributions over $N^{\prime}<N$ options (see (2.7)), it follows for any cost function $C$ that the function $f$ defined by f(N):=C\big{(}\tfrac{1}{N},\dots,\tfrac{1}{N}\big{)} is monotonically increasing. Therefore the claim follows directly from Shannon’s proof [85], where he shows that this monotonicity, additivity under coarse-graining, and continuity determine Shannon entropy up to a constant factor. ∎

Proposition A.3** (Prop.2.5).**

The uniform distribution $(\tfrac{1}{N},\dots,\tfrac{1}{N})$ is the unique minimal element in $\mathbb{P}_{\Omega}$ with respect to $\prec$, i.e.

[TABLE]

Proof.

For the proof of (A.4), let $(\Pi_{i})_{i=1}^{N}$ be the family of all cyclic permutations of the $N$ entries of a probability vector $p$, such that

[TABLE]

It follows that $\sum_{i=1}^{N}\Pi_{i}(p)=e$ for all $p\in\mathbb{P}_{\Omega}$, where $e=(1,\dots,1)$ as above, and therefore the uniform distribution $\frac{1}{N}e$ is given by a convex combination of permutations of $p$, so that $(iv)$ in Theorem 2.3 implies $\frac{1}{N}e\prec p$. There is many different ways to prove uniqueness. A direct way is to assume there exists $q\in\mathbb{P}$ with $q\prec\frac{1}{N}e$ for all $p\in\mathbb{P}_{\Omega}$, so that by $(iii)$ in Theorem 2.3 there exists a stochastic matrix $A$ with $\frac{1}{N}eA=q$. But since $eA=e$ ($A$ stochastic), it follows that $q=\frac{1}{N}e$. An indirect way would be to use that if $q\prec p$ for all $p\in\mathbb{P}_{\Omega}$, then from Example Example we know that this implies $H(q)\geq H(p)$ for all $p\in\mathbb{P}_{\Omega}$, where $H$ denotes the Shannon entropy, simply because $-H$ is a cost function. In particular, $q$ maximizes $H$ and therefore coincides with the uniform distribution $\frac{1}{N}\,e$. ∎

Proposition A.4** (Equivalence of $(i),(iii),(v)$ in Theorem 2.7).**

The following are equivalent

1. $(i)$

$p^{\prime}\prec_{q}p$, i.e. $p^{\prime}$ contains more uncertainty relative to $q$ than $p$ (Def. 2.6)

** 2. $(iii)$

$p^{\prime}=pA$* for a $q$-stochastic matrix $A$, i.e. $Ae=e$ and *$qA=q$

** 3. $(v)$

$\sum_{i=1}^{l-1}(p^{\prime}_{i})^{\downarrow}+a_{q}(k,l)(p^{\prime}_{l})^{\downarrow}\ \leq\ \sum_{i=1}^{l-1}p_{i}^{\downarrow}+a_{q}(k,l)p_{l}^{\downarrow}$* for all $\alpha\sum_{i=1}^{l-1}q_{i}^{\downarrow}\leq k\leq\alpha\sum_{i=1}^{l}q_{i}^{\downarrow}$ and $1\leq l\leq N$, where $a_{q}(k,l)\coloneqq(\frac{k}{\alpha}-\sum_{i=1}^{l-1}q_{i}^{\downarrow})/q_{l}^{\downarrow}$, and the arrows indicate that $(p_{i}^{\downarrow}/q_{i}^{\downarrow})_{i}$ is ordered decreasingly.*

Proof.

We will make use of the fact that $\Lambda_{q}:\mathbb{P}_{\Omega}\to\mathbb{P}_{\tilde{\Omega}}$ has a left inverse $\Lambda_{q}^{-1}:\Lambda_{q}(\mathbb{P}_{\Omega})\to\mathbb{P}_{\Omega}$ satisfying $\Lambda_{q}^{-1}\Lambda_{q}=\mathbb{I}$, where $\mathbb{I}$ denotes the identity on $\mathbb{P}_{\Omega}$. This can be verrified by simply multiplying the corresponding matrices, given by

[TABLE]

and

[TABLE]

and noting that, by definition, $\alpha\,q_{i}=|A_{i}|$.

Characterization $(v)$ follows from $(vi)$ of Theorem 2.3 and Definition 2.6, since $p^{\prime}\prec_{q}p$ if and only if

[TABLE]

for all $1\leq k\leq{\alpha-1}$, from which $(v)$ is an immediate consequence.

$(i)\Rightarrow(iii)$: Assuming that $p^{\prime}\prec_{q}p$ we have $\Lambda_{q}p^{\prime}\prec\Lambda_{q}p$ and therefore, by $(iii)$ in Theorem 2.3, there exists a doubly stochastic matrix $B$ such that $\Lambda_{q}p^{\prime}=B^{T}\Lambda_{q}p$. With $A^{T}\coloneqq\Lambda_{q}^{-1}B^{T}\Lambda_{q}$ it follows that

[TABLE]

where we have used that $(\Lambda_{q}^{-1})^{T}e=e$ and $\Lambda_{q}^{T}e=e$ which is easy to check, and $Be=e$ from $B$ being a stochastic matrix. Note that, by slightly abusing notation, here $e$ is always the constant vector $(1,\dots,1)$ irrespective of the number of its entries ($N$ or $\alpha$, depending on which space the operator is defined). Moreover, we have

[TABLE]

where we have used that $\Lambda_{q}q$ by definition is the uniform distribution on $\mathbb{P}_{\tilde{\Omega}}$, i.e. $\Lambda_{q}q=\frac{1}{\alpha}\,e$, and therefore also $\Lambda_{q}^{-1}e=\alpha\,q$. In particular, since also $A_{ij}\geq 0$ ($B,\Lambda_{q},\Lambda_{q}^{-1}$ have non-negative entries), it follows that $A$ is a $q$-stochastic matrix such that $p^{\prime}=A^{T}p=pA$.

$(i)\Leftarrow(iii)$: Similarly, if $A$ is a $q$-stochastic matrix with $p^{\prime}=pA$, then $\Lambda_{q}p^{\prime}=B^{T}\Lambda_{q}p$, where $B\coloneqq(\Lambda_{q}^{-1})^{T}A\Lambda_{q}^{T}$ satisfies $B^{T}e=\alpha\,\Lambda_{q}A^{T}q=\alpha\,\Lambda_{q}q=e$ and $Be=(\Lambda_{q}^{-1})^{T}Ae=(\Lambda_{q}^{-1})^{T}e=e$, where we have used that $\Lambda_{q}^{-1}e=\alpha q$, $\Lambda_{q}q=\frac{1}{\alpha}\,e$, $\Lambda_{q}^{T}e=e$, and $(\Lambda_{q}^{-1})^{T}e=e$. In particular, since also $B_{ij}\geq 0$, $B$ is doubly stochastic and therefore $\Lambda_{q}p^{\prime}\prec\Lambda_{q}p$, i.e. $p^{\prime}\prec_{q}p$. ∎

Proposition A.5** (Prop. 2.8).**

The prior $q\in\mathbb{P}_{\Omega}$ is the unique minimal element in $\mathbb{P}_{\Omega}$ with respect to $\prec_{q}$, that is $q\prec_{q}p$ for all $p\in\mathbb{P}_{\Omega}$.

Proof.

Let $p\in\mathbb{P}_{\Omega}$, and let $P:=\Lambda_{q}p$ denote its representation in $\mathbb{P}_{\tilde{\Omega}}$. Then $Q\prec P$ by Proposition A.3 (uniform distributions are minimal) and therefore $q=\Lambda_{q}^{-1}Q\prec_{q}p$, in particular $q$ is minimal with respect to $\prec_{q}$. For uniqueness, let $p^{\prime}$ be possibly another minimal element. Then $p^{\prime}\prec_{q}q$ and therefore by $(iii)$ in Theorem 2.7 there exists a $q$-stochastic matrix $A$ with $p^{\prime}=qA$. But since $A$ is $q$-stochastic, $qA=q$, and thus $p^{\prime}=q$. ∎

Proposition A.6** (Example Example: Superadditivity of $f$-divergences under coarse-graining).**

All relative cost functions of the form

[TABLE]

with $f$ (strictly) convex and differentiable on $[0,1]$, and $f(1)=0$, are superadditive with respect to coarse-graining, that is, for $Z=(X,Y)$,

[TABLE]

whenever $C_{q}(X)\coloneqq C_{q(X)}(p(X))$ and $C_{q}(Y|X)\coloneqq\mathbb{E}_{p(X)}[C_{q(Y|X)}(p(Y|X))]$, which for cost functions that are symmetric with respect to permutations of $((p_{i},q_{i}))_{i=1,\dots,N}$ (such as (2.20)) is equivalent to

[TABLE]

Proof.

This is a simple corollary to Proposition A.1, after establishing the following interesting property of cost functions of the form (A.5):

[TABLE]

where $\Lambda_{q}$ denotes the representation mapping defined in Section 2.3 that maps $q$ to a uniform distribution on an elementary decision space $\tilde{\Omega}$ of $|\tilde{\Omega}|=\alpha$ elements, given by $(\Lambda_{q}p)(\omega)=\frac{1}{\alpha}\frac{p_{i}}{q_{i}}$ whenever $\omega\in A_{i}$, where $\{A_{i}\}_{i=1}^{N}$ is a disjoint partition of $\tilde{\Omega}$ such that $|A_{i}|=\alpha\,q_{i}$. Equation (A.7) then follows from

[TABLE]

where we have used that $\sum_{w\in A_{i}}1=|A_{i}|=\alpha q_{i}$. Hence the case of a non-uniform prior reduces to the case of a uniform prior, which was shown in Proposition A.1. ∎

Proposition A.7** (Example Example: Characterization of Kullback-Leibler divergence).**

If $C_{q}$ is a continuous cost function relative to $q$ that is additive under coarse-graining, that is $C_{q}(Z)=C_{q}(X)+C_{q}(Y|X)$ in the notation of Proposition A.6, then

[TABLE]

for some $\alpha\geq 0$, where $D_{KL}(p\|q)$ denotes the Kullback-Leibler divergence $D_{KL}(p\|q)=\sum_{i}p_{i}\log(p_{i}/q_{i})$.

Proof.

First, we show that any relative cost function that is additive under coarse-graining satisfies (2.22), the monotonicity property for uniform distributions: If $f(M,N)$ denotes the cost $C_{u_{N}}(u_{M})$ of a uniform distribution $u_{M}$ over $M$ elements relative to a uniform distribution $u_{N}$ over $N\geq M$ elements, then

[TABLE]

Once (A.9) has been established, then (A.8) goes back to a result by Hobson [40] (see also [63]), whose proof is a modification of Shannon’s axiomatic characterization [85].

The first property in (A.9) actually is true for all relative cost functions: For $q=u_{N}$ with $N=|\Omega|$, we have $p^{\prime}\prec_{q}p$ iff $p^{\prime}\prec p$ and thus the first property follows from (2.7), and the same is true in the case when $N<|\Omega|$, since we always assume that $p^{\prime},p$ are absolutely continuous with respect to $q$, which allows to redefine $\Omega$ to only contain the $N$ options covered by $q$.

For the proof of the second property in (A.9), we let the random variable $X$ indexing the partitions $E_{1},E_{2}$ of $\Omega$, where $E_{1}$ denotes the support of $u_{N^{\prime}}$ and $E_{2}=\Omega{\setminus}E_{1}$ its complement, and $Y$ representing the choice inside of the selected partition $E_{i}$ given $X=i$. Letting $q(Z)=u_{N}$, and $p(Z)=u_{N^{\prime}}$, then it follows from addivity under coarse-graining that $C_{q(X)}(p(X))=C_{u_{N}}(u_{N^{\prime}})=f(N^{\prime},N)$, and letting $p(Z)=u_{M}$, we obtain

[TABLE]

since $p(X{=}1)=1$, $p(X{=}2)=0$, and $C(Y|X=1)=C_{u_{N^{\prime}}}(u_{M})$. Therefore it follows that $f(M,N)\geq f(M,N^{\prime})$. ∎

Proposition A.8** (Prop. 3.2).**

If $(p_{\beta})_{\beta\geq 0}$ is a family of bounded-optimal posteriors given by (3.3) with cost function $C_{q}(p)=D_{KL}(p\|q)$, then $\beta$ is a resource parameter with respect to any cost function, in particular

[TABLE]

Proof.

Part of the proof generalizes a result in [78] to the case of a non-uniform prior $q$, by making use of our new characterization $(v)$ of $\prec_{q}$, by which it suffices to show that $\beta\mapsto\sum_{i=1}^{l-1}(p_{\beta,i})^{\downarrow}+a_{q}(k,l)(p_{\beta,l})^{\downarrow}$ is an increasing function of $\beta$ for all $k,l$ specified in Theorem 2.7. By (3.6) below, we have

[TABLE]

where $U_{i}$ are the decreasingly arranged utility values $U(x)$ for $x\in\Omega$, so that also $p_{\beta,i}/q_{i}$ is arranged decreasingly. From the ordering of the $U_{i}$, it is easy to see that

[TABLE]

with the notation $p_{k}\coloneqq p_{\beta,k}$, from which it follows that $S_{k}\coloneqq\sum_{i=1}^{k}\hat{p}_{k}U_{i}$ with $\hat{p}_{k}\coloneqq{p_{k}}/{\sum_{j=1}^{k}p_{j}}$, is monotonically decreasing in $k$ (with $S_{N}=\mathbb{E}_{p}[U]$), and therefore

[TABLE]

for all $k\leq N$. Hence, it suffices to show that $\sum_{i=1}^{l-1}x_{i}+tx_{l}\geq 0$ if $\sum_{i=1}^{k}x_{i}\geq 0$ for all $k$ and $t\in[0,1]$. If $x_{l}\geq 0$, there is nothing to show, and if $x_{l}<0$, we have $\sum_{i=1}^{l-1}x_{i}+tx_{l}\geq\sum_{i=1}^{l}x_{i}\geq 0$, which completes the proof of $p_{\beta^{\prime}}\prec_{q}p_{\beta}$.

It remains to see that $p_{\beta}\not\prec_{q}p_{\beta^{\prime}}$. This follows again from $(v)$ in Theorem 2.7, more precisely from the requirement that $p_{\beta,1}\leq p_{\beta^{\prime},1}$ if $p_{\beta}\prec_{q}p_{\beta^{\prime}}$, after establishing that $\beta\mapsto Z_{\beta}^{-1}e^{\beta U_{1}}$ is monotonically increasing. For the latter, note that since the $U_{i}$ are ordered decreasingly, $\sum_{i=1}^{N}U_{i}q_{i}e^{\beta U_{i}}\leq U_{1}\sum_{i=1}^{N}q_{i}e^{\beta U_{i}}$ from which follows that $\partial_{\beta}(Z_{\beta}^{-1}e^{\beta U_{1}})\geq 0$, which completes the proof. ∎

Acknowledgement

This study was funded by the European Research Council (ERC-StG-2015-ERC Starting Grant, Project ID: 678082, “BRISC: Bounded Rationality in Sensorimotor Coordination”).