An information theoretic model for summarization, and some basic results

TL;DR

This paper introduces an information theoretic framework for summarization, modeling reports as binary object sets and optimizing subset selection to minimize semantic loss, with results for known and unknown report distributions.

Contribution

It formulates a novel information theoretic model for summarization and derives methods to minimize semantic loss under different distribution knowledge scenarios.

Findings

Optimal summarizers minimize semantic loss when distribution is known.

Summarizers approximate minimal semantic loss when distribution is unknown.

The model provides a theoretical basis for evaluating summarization quality.

Abstract

A basic information theoretic model for summarization is formulated. Here summarization is considered as the process of taking a report of $v$ binary objects, and producing from it a $j$ element subset that captures most of the important features of the original report, with importance being defined via an arbitrary set function endemic to the model. The loss of information is then measured by a weight average of variational distances, which we term the semantic loss. Our results include both cases where the probability distribution generating the $v$-length reports are known and unknown. In the case where it is known, our results demonstrate how to construct summarizers which minimize the semantic loss. For the case where the probability distribution is unknown, we show how to construct summarizers whose semantic loss when averaged uniformly over all possible distribution converges to the minimum.

Equations

$$|{\mathcal{T}}_{(x^n)}^n|$$

$$|{\mathcal{P}}_n(\mathcal{X})|$$

$$1_{\hat{\mathcal{X}}}(x)=\{\begin{array}{ll}1& \text{ if }x\in \hat{\mathcal{X}}\\ 0& \text{ o.w. }\end{array},$$

$$\mathcal{X}(y)\triangleq \{x\in \mathcal{X}:y\subset x\}.$$

$$i_y(x)\triangleq 1_{\mathcal{X}(y)}(x)\frac{p(x)}{p(\mathcal{X}(y))}.$$

$$\ell (X;Y|u)$$

$$\mathcal{Y}=\left(\begin{array}{c}\mathcal{V}\\ j\end{array}\right)\times \{0,1{\}}^j$$

$$\ell (X(1);Y|u)\le \delta$$

$${\int }_{\mathcal{P}(\mathcal{X})}\ell (X(1);Y|u)\mathrm{d}r\le \delta$$

$$\sum _{\begin{array}{c}x\in \mathcal{X},\\ y\in \mathcal{Y}\end{array}}p(x)s_x(y)\sum _{\mathcal{W}\subseteq \mathcal{X}}u(x,\mathcal{W})(1-i_y(\mathcal{W})),$$

$$\sum _{x}p(x)\underset{y\in \mathcal{Y}}{\min }\sum _{\mathcal{W}\subseteq \mathcal{X}}u(x,\mathcal{W})(1-i_y(\mathcal{W})).$$

$$\sum _{\mathcal{W}}u(x(1),\mathcal{W})\left(1-\frac{p(\mathcal{W}\cap \mathcal{X}(y)}{p(\mathcal{X}(y))}\right).$$

$$\sum _{\begin{array}{c}{x}^n\in {\mathcal{X}}^n,\\ y\in \mathcal{Y},\\ \mathcal{W}\subseteq \mathcal{X}\end{array}}\frac{s_{x^n}(y)u(x(1),\mathcal{W})}{|{\mathcal{T}}_{(x^n)}^n||{\mathcal{P}}_n(\mathcal{X})|}\left(1-{\eta }_{x^n,y,\mathcal{W}}\right)$$

$${\eta }_{x^n,y,\mathcal{W}}$$

$$\cdot \sum _{k=0}^{\infty }\frac{\left(n+|\mathcal{X}|-(n+|\mathcal{X}|+1){\hat{q}}^{(x^n)}\left(\mathcal{X}(y)\right)+k\right)!\left(n+|\mathcal{X}|\right)}{\left(n+|\mathcal{X}|-(n+|\mathcal{X}|+1){\hat{q}}^{(x^n)}(\mathcal{X}(y))\right)!\left(n+|\mathcal{X}|+k\right)!},$$

$$q^{(x^n)}(a)$$

$${\hat{q}}^{(x^n)}(a)$$

$$\frac{c+1}{c-b}\le \sum _{t=0}^{\infty }\frac{(b+t)!c!}{(c+t)!b!}\le \frac{c+1}{c-b}\left(1+\epsilon (c-b)\right)$$

$$\epsilon (a)=3\frac{1+\ln (a)}{a}+4e^{\frac{1}{12}}\cdot 2^{-\frac{a}{2}}.$$

$$-{\lambda }_n+\sum _{\begin{array}{c}{x}^n\in {\mathcal{X}}^n\end{array}}\frac{\mu (x^n)}{|{\mathcal{T}}_{(x^n)}^n||{\mathcal{P}}_n(\mathcal{X})|}$$

$$\mu (x^n)\triangleq \underset{y\in \mathcal{Y}}{\min }\sum _{\mathcal{W}\subseteq \mathcal{X}}u(x(1),\mathcal{W})\left(1-\frac{q^{(x^n)}(\mathcal{W}\cap \mathcal{X}(y))}{{\hat{q}}^{(x^n)}(\mathcal{X}(y))}\right)$$

$$0<{\lambda }_n<u^{\ast }\left[\frac{(|\mathcal{X}|-1)|\mathcal{X}|}{n-\sqrt{n}+|\mathcal{X}|-1}\epsilon (|\mathcal{X}|2^{-j})+\epsilon (\sqrt{n})\right]$$

$$\sum _{\mathcal{W}\subseteq \mathcal{X}}u(x(1),\mathcal{W})\left(1-\frac{q^{(x^n)}(\mathcal{W}\cap \mathcal{X}(y))}{{\hat{q}}^{(x^n)}(\mathcal{X}(y))}\right)$$

$$u(x,\mathcal{W})=\{\begin{array}{ll}1& \text{ if }\mathcal{W}=\mathcal{X}(y_x)\\ 0& \text{otherwise}\end{array}.$$

$${\int }_0^c(c-x{)}^a{x}^b\mathrm{d}x=\frac{a!b!}{(a+b+1)!}{c}^{a+b+1}$$

$${\int }_0^c(c-x{)}^a{x}^b\mathrm{d}x$$

$$=c^{a+b+1}{\int }_0^1(1-t{)}^a{t}^b\mathrm{d}t.$$

$${\int }_0^1(1-t{)}^a{t}^b\mathrm{d}t$$

$$={\left[\frac{1}{b+1}(1-t{)}^a{t}^{b+1}\right]}_0^1+\frac{a}{b+1}{\int }_0^1(1-t{)}^{a-1}{t}^{b+1}\mathrm{d}t$$

$$=\frac{a}{b+1}{\int }_0^1(1-t{)}^{a-1}{t}^{b+1}\mathrm{d}t$$

$${\int }_0^1(1-t{)}^a{t}^b\mathrm{d}t=\frac{b!a!}{(a+b+1)!},$$

$$\left\{(p_{1},\dots,p_{|\mathcal{X}|})\in\mathbb{R}^{|\mathcal{X}|}:\left(\begin{array}[]{l}p_{i}\in\left[0,\bar{p}_{i}\right]\\ ~{}~{}~{}~{}i\in\left\{1,\dots,|\mathcal{X}|-1\right\},\\ p_{|\mathcal{X}|}=\bar{p}_{|\mathcal{X}|}\end{array}\right)\right\},$$

$${\int }_0^1{\int }_0^{{\bar{p}}_2}\cdots {\int }_0^{{\bar{p}}_{|\mathcal{X}|-1}}c\mathrm{d}{p}_1\dots \mathrm{d}{p}_{|\mathcal{X}|-1}=1.$$

Full text

Our results include both cases where the probability distribution generating the $v$-length reports are known and unknown. In the case where it is known, our results demonstrate how to construct summarizers which minimize the semantic loss. For the case where the probability distribution is unknown, we show how to construct summarizers whose semantic loss when averaged uniformly over all possible distribution converges to the minimum.

I Introduction

For a concrete example of how we shall define information summarization, consider the following weather report.

[TABLE]

Our stance is that such a report is overly detailed, and wish to design a system that produces summaries such as the following.

[TABLE]

In this example it is important to note a typhoon already implies high winds, heavy rain and low visibility, and heavily implies the absence of snow. At the same time, the presence of a typhoon does not generally indicate a high UV index or the lack of smog; these events should still be reported.

In the abstract, the goal of summarization is to reduce the dimension of data, without excessive loss of “information.” This abstract notion is very similar to that of compression and rate-distortion theory [1, Chapter 5, 10]; summarization is distinguished in two important ways. First, unlike compression and rate distortion which feature both an encoder to construct the efficient representation and a decoder to interpret this representation, a summarizer only has a single element, the output of which should be ready for immediate consumption. Second, we must take into account the importance of the underlying information, as opposed to simply the likelihood. For instance, smog may be less likely than a typhoon, but the typhoon is more essential to include given that both occur.

Despite similarities to established concepts in information theory, to the best of our knowledge, summarization has never been considered from the information theoretic perspective. Instead most of the literature exists within the natural language processing community and the machine learning community, see [2, 3, 4] and references there within. The approach of these communities is to directly engage the general problem, searching for efficient practical solutions which provide empirically good results111For discussion on this viewpoint, see Simeone [5, Chapter 1].. This differs from a traditional information theoretic approach, where a simplified model is established and analyzed in order to determine fundamental limits of the operational parameters, and to gain insight into how to achieve those limits222For discussion on this viewpoint, see Han [6, Preface]..

To simplify this model, we shall make the following assumptions. First, the data to be summarized is a length-$v$ binary sequence, which has an arbitrary (not necessarily independent) probability distribution relating each symbol. While the probability distribution over a length-$v$ binary sequence is arbitrary, every length-$v$ sequence the summarizer observes is independent and identically distributed. Second, we assume that the summarizers output needs be “extractive,” meaning that the summarizer can only produce a subset of what is input, as in the weather example. Finally, we assume the existence of an arbitrary set function that can be used to measure the “semantic information” of a random variable. This last assumption will be further justified in Section III via example, but it is worth mentioning that, as shown by Yeung [7], Shannon’s measure of (nonsemantic) information (entropy) has such a representation. Spurred by this, Lin and Bilmes [3, 4] have recently argued for the use of submodular functions in an effort to axiomatically define a notion of semantic information. Regardless, we will make no other assumptions on this function other than existence and that it is finite and positive.

II Notation

Random variables (RV(s)) will be written in upper case, constants in lower case, and sets in calligraphic font. For example $X$ can take on value $x$ from $\mathcal{X}.$ A $n$-length sequence of random variables, variables or sets will be denoted with the power $n$, such as $X^{n}$. Among sets $\mathcal{P}(\mathcal{X}|\mathcal{Y})$ will hold special meaning as the set of conditional probability distributions on the set $\mathcal{X},$ when given a value from the set $\mathcal{Y}$. That is, if $p\in\mathcal{P}(\mathcal{X}|\mathcal{Y})$, then $p_{y}(x)\in[0,1]$ and $\sum_{x\in\mathcal{X}}p_{y}(x)=1$ for all $x\in\mathcal{X},~{}y\in\mathcal{Y}$. For convenience, given $\mathcal{W}\subset\mathcal{X}$ then $\mathcal{P}(\mathcal{W})\subset\mathcal{P}(\mathcal{X})$, where only symbols in $\mathcal{W}$ have non-zero probability. The symbol $\sim$ will be used to relate probability distributions and random variables. For example if $X\sim p(x)$ and $Y\sim q_{X}(y)$, for some $q\in\mathcal{P}(\mathcal{Y}|\mathcal{X})$ and $p\in\mathcal{P}(\mathcal{X})$, then $\Pr\left(X=x,Y=y\right)=q_{x}(y)p(x)$. When a set is used in a probability distribution, such as $p(\mathcal{\hat{X}})$ for some $\mathcal{\hat{X}}\subset\mathcal{X}$ and $p\in\mathcal{P}(\mathcal{X}),$ it means $\sum_{x\in\mathcal{\hat{X}}}p(x).$

We shall use $\pi^{(x^{n})}$ to denote the empirical distribution of a series $x^{n}$, for example $\pi^{(0,1,1,1)}(0)=\frac{1}{4}$ while $\pi^{(0,1,1,1)}(1)=\frac{3}{4}.$ The set $\mathcal{T}^{n}_{(x^{n})}$ denotes the set of $n$-length sequences with the same empirical distribution as $x^{n}$, that is $\mathcal{T}^{n}_{(x^{n})}\triangleq\{\hat{x}^{n}:\pi^{(\hat{x}^{n})}=\pi^{(x^{n})}\}.$ It will be important to note that

[TABLE]

Next $\mathcal{P}_{n}(\mathcal{X})$ denotes the set of valid empirical distributions for $n$-length sequences of symbols from $\mathcal{X}$. Again it is important to note

[TABLE]

$1_{\mathcal{\hat{X}}}$ is the indicator function,

[TABLE]

and $\mathbb{E}$ is the expected value operator, where the expectation is taken over all random variables.

III System Model and Justification

The objects to be summarized will be referred to as reports. A sequence of $n$-reports will be denoted by a sequence of RVs $X^{n}=(X(1),\dots,X(n))\sim\prod_{i=1}^{n}p(x(i))$, where $p\in\mathcal{P}(2^{\mathcal{V}})$ and $\mathcal{V}$ is the finite set of possible events. Without loss of generality we will assume $\mathcal{V}=\left\{1,\dots,v\right\}$ for some positive integer $v$, and accordingly will refer to $X$ as a $v$-symbol binary sequence with $x_{j}=1$ denoting possible event $j\in\mathcal{V}$ occurring for report $x$. From here forward $\mathcal{X}\triangleq 2^{\mathcal{V}},$ or the power set of $\mathcal{V}$, for convenience.

Although given $n$-reports the summarizer only needs to summarize $X(1)$, as shown in Figure 2; to summarize is to produce a subset of $j\in\mathbb{Z}^{+}$ possible events and indicate whether or not they each occurred. Formally, the summarizer produces $Y=(\hat{Y},\tilde{Y})$, $\mathcal{\hat{Y}}=\left(\begin{matrix}\mathcal{V}\\ j\end{matrix}\right)$, $\mathcal{\tilde{Y}}=\{0,1\}^{j}$, where $\hat{Y}$ is a subset containing $j$ possible events and $\tilde{Y}$ is the indication of whether or not the possible events in $\hat{Y}$ occurred. Because the summarizer only needs to summarize the first report, we will refer to $X(1)$ as the current report, and $X_{2}^{n}\triangleq(X(2),\dots,X(n))$ as the report history. Note that finding the optimal summary for $X(1)$ also finds the optimal summary algorithm for $X(k)$ for $k=\{2,\dots,n\}$ since they are identically distributed.

Notice that a summary does not necessarily provide all of the information about the report, and moreover there are multiple reports for which a given summary may be the representative. A specific summary $y$ may be the summary for any report in333We write $y\subset x$ if a summary $y$ could lead to $x.$ That is, if $\hat{y}=\{1,2,5\}$ then $y=(\hat{y},\tilde{y})\subset x$ if and only if $\tilde{y}=(x_{1},x_{2},x_{5})$.

[TABLE]

Clearly for a given summarization algorithm, each $x\in\mathcal{X}(y)$ does not necessarily generate summary $y$.

To relate the output of the summarizer to the input we introduce a conditional probability distribution called the summary interpretation,

[TABLE]

The summary interpretation is equal to the probability of a report for a given summary when averaged uniformly over all possible summarizers. In this way it represents an end user which has absolutely no information about the summarizing algorithm, but knows perfectly the distribution of what is being summarized.

Having related the report and the summary, our goal will be to produce summaries which capture most of the “semantic information” of the current report. In order to capture “semantic information,” included in the model will be a set function $u:\mathcal{X}\times 2^{\mathcal{X}}\rightarrow\mathbb{R}^{+}$ denoting semantic weights. The semantic weights assign to each pair of $x\in\mathcal{X}$ and subset $\mathcal{W}\subseteq\mathcal{X}$ a number representative of the importance that the summary convey $x\in\mathcal{W}$ (if $x\notin\mathcal{W}$ then $u(x,\mathcal{W})=0$). The motivation behind the semantic weights is best viewed by the weather example. With each report there are a number of different possible implications, for instance, the report might imply extra layers of clothing are needed, or that it is a nice day to go outside, or that serious meteorological event is occurring. Each of these implications can be considered semantic information, as it has some intuitive meaning. Furthermore, each of these implications is valid for a large number of possible reports. In that sense, each set $\mathcal{W}$ is representative of some semantic meaning shared between the collective of reports in the set, and $u(x,\mathcal{W})$ is represents how important this semantic meaning is to the report $x$.

Having defined all aspects endemic to the model, we now move to discussing the operational parameters. To aggregate and measure the performance of the summarizer, we shall use the semantic loss, which is the summarizer analog to a distortion criteria in rate distortion theory.

Definition 1**.**

The semantic loss of $X$ to $Y$ with semantic weights $u:\mathcal{X}\times 2^{\mathcal{X}}\rightarrow\mathbb{R}^{+}$ is

[TABLE]

Consider the semantic loss when there is a single $\mathcal{W}$ such that $u(x,\mathcal{W})\neq 0.$ In this case the semantic loss is the variational distance between the summary interpretation and the closest distribution such that only reports in $\mathcal{W}$ occur. Clearly, if only reports in $\mathcal{W}$ were possible, given a particular summary, then this summary would losslessly convey that a report in $\mathcal{W}$ occurred. Using an $f$-divergence (see [8, Chapter 4]), namely variational distance, give us a well studied way to then measure the distance between the summary interpretation and the convex set of distributions which perfectly convey $\mathcal{W}$. This distance is then averaged over all semantic meanings according to the semantic weights.

We conclude the section with a more formal definition of a summarizer. For the purpose of easily specifying operational parameters, we shall refer to a summarizer by the probability distribution relating the summary $Y$ and the reports $X^{n}$.

Definition 2**.**

For each $j\in\mathbb{Z}^{+}$ and $\delta\in\mathbb{R}^{+}$, a summarizer $s\in\mathcal{P}(\mathcal{Y}|\mathcal{X}^{n})$ has length $j$ if

[TABLE]

and has $\delta$ semantic loss for reports $X^{n}$ and semantic weights $u$ if

[TABLE]

for $Y\sim s_{X^{n}}(y).$

III-A Universal summarization

In the universal setting, the summarizer is no longer aware of the distribution $p\in\mathcal{P}(\mathcal{X})$ by which $X^{n}\sim\prod_{j=1}^{n}p(x(j)),$ Since we still assume the end user is aware of this distribution, the summary interpretation remains unchanged. But, as our results demonstrate, knowing the summary interpretation is of vital importance to cultivating good summarizers. Since the summary interpretation is no longer known, the summarizer must be able to adapt itself based upon the report history.

To measure the performance in this case, we will consider the semantic loss when averaged uniformly over all possible distributions of $P(\mathcal{X})$. In that way, we can ensure that the number of distributions for which the summarizer performs poorly are relatively small.

Definition 3**.**

A summarizer $s\in\mathcal{P}(\mathcal{Y}|\mathcal{X}^{n})$ has $\delta$-uniform average semantic loss for semantic weights $u$ if

[TABLE]

for $Y\sim s_{X^{n}}(y),$ $X^{n}\sim\prod_{j=1}^{n}r(x(j))$ and $r$ uniform over $\mathcal{P}(\mathcal{X}).$

IV Results

.

Our objective is to find optimal, or close to optimal, summarization algorithms. To this end, we first classify what the semantic loss is for a summarizer, and then use that value to determine which summarizer produces the smallest value.

Lemma 4**.**

Summarizer $s\in\mathcal{P}(\mathcal{Y}|\mathcal{X}^{n})$ has a semantic loss for reports $X^{n}\sim\prod_{m=1}^{n}p(x(m))$ and semantic weights $u$ of

[TABLE]

where $s_{x}(y)=\sum_{x^{n}:x(1)=x}\left(\prod_{m=2}^{n}p(x(m))\right)s_{x^{n}}(y)$.

Corollary 5**.**

The minimum semantic loss for reports $X^{n}\sim\prod_{m=1}^{n}p(x(m))$ and semantic weights $u$ is

[TABLE]

See Appendix B for proof.

Lemma 4 demonstrates that the semantic loss is the weighted average of the summary interpretation’s concentration outside $\mathcal{W}$; that is the semantic loss is the weighted average of the various semantic meanings being false under the summary interpretation. Corollary 5 also suggests a summarization algorithm to achieve it. In particular, given reports $x^{n}$, the summarizer selects the summary $y$ which minimizes

[TABLE]

To do so though, requires the summarizer know $p$ a priori.

When moving to the universal setting the value of $p$ is unknown, and instead the distribution $p$ has to be inferred from the reports. Here we seek to derive the uniform average semantic loss, for semantic weights $u$, and then find the summarization algorithm to optimize it.

Theorem 6**.**

Summarizer $s\in\mathcal{P}(\mathcal{Y}|\mathcal{X}^{n})$ has a uniform average semantic loss for semantic weights $u$ of

[TABLE]

where

[TABLE]

See Appendix C for proof.

Theorem 6 though, unlike Lemma 4, is not a closed form solution. In order to assuage this malady the following approximation is provided.

Lemma 7**.**

For positive integers $b,~{}c$ such that $1\leq b<b+2\leq c$,

[TABLE]

where

[TABLE]

See Appendix D for proof.

Using Theorem 6 and Lemma 7 will allow us to construct a theorem analogous to Corollary 5.

Theorem 8**.**

The minimum uniform average semantic loss for semantic weights $u$, is equal to

[TABLE]

where

[TABLE]

and $q^{(x^{n})}$ and $\hat{q}^{(x^{n})}$ from Theorem 6, while $\lambda_{n}$ satisfies

[TABLE]

with $u^{*}=\sum_{x\in\mathcal{X}}|\mathcal{X}|^{-1}\sum_{\mathcal{W}\subseteq\mathcal{X}}u(x,\mathcal{W}),$ and $\varepsilon$ from Lemma 7.

See Appendix E for proof.

Note that $\lim_{n\rightarrow\infty}\lambda_{n}=0$, and thus Theorem 8 shows that a summarizer that $y$ to minimize

[TABLE]

will asymptotically with report history minimize the uniform average semantic loss for semantic weights $u$. Hence regardless of the set function to characterize semantic meaning, the optimal summarizer still treats the underlying summary interpretation as $\frac{q^{(x^{n})}}{\hat{q}^{(x^{n})}}.$

V Conclusion

Going forward it will be important to derive representations for the semantic weights which are practical and perform well in practice. Indeed, one aspect not previously mentioned is that for any “optimal” summary, regardless of how optimal is defined, there are a set of semantic weights such that it is also the optimal summary in our model. To see this, consider an optimal (deterministic) summarizer defined by the mapping $x\mapsto y_{x}$, and recognize that this is also an optimal summary in our model for semantic weights

[TABLE]

While the above is clearly not edifying, it does demonstrate the generality of our model. Nevertheless, determination of simple semantic weights that perform well in practice would validate the presented model.

Appendix A Lemmas

Lemma 9**.**

[TABLE]

for all non negative integers $a,b$ and real number $c\in[0,1]$.

Proof:

First observe that

[TABLE]

The final integral can be found in [1], but we include it for completeness. Specifically,

[TABLE]

by using integration by parts (setting $u=(1-t)^{a}$ and $\mathrm{d}v=t^{b}\mathrm{d}t$). Thus

[TABLE]

from recursion. ∎

Corollary 10**.**

Distribution $r\in\mathcal{P}(\mathcal{P}(\mathcal{X}))$, where $r(p)=(|\mathcal{X}|-1)!$, is the uniform distribution over $\mathcal{P}(\mathcal{X})$.

Proof:

First note that $\mathcal{P}(\mathcal{X})$ is a convex set

[TABLE]

where $\bar{p}_{i}\triangleq 1-\sum_{j=1}^{i-1}p_{j}.$ Hence if $r$ is the uniform probability density function over $\mathcal{P}(\mathcal{X})$ then there exists a constant positive real number $c$ such that $r(p)=c$ and

[TABLE]

Using Lemma 9 to repeatedly evaluate the LHS of (2) yields

[TABLE]

hence

[TABLE]

∎

Lemma 11**.**

[TABLE]

for all positive integers $k$ and real numbers $t>1$.

Proof:

First note that $k^{-t}$ is convex in $k$ since $t>1$, and thus

[TABLE]

∎

Lemma 12**.**

For $a>b$ positive real numbers,

[TABLE]

Proof:

For any positive real number $y$, by definition

[TABLE]

Hence we must show that $f(x,y)\triangleq\left(1+\frac{y}{x}\right)^{x}$ is a monotonically increasing function of $x$ for all $x\geq 1$, since then

[TABLE]

Clearly though

[TABLE]

This derivative is always positive, since the function $g(u)\triangleq\ln u+\frac{1}{u}-1\geq 0$ for all $u\geq 1$. Indeed $g(u)\geq g(1)$ since

[TABLE]

and $g(1)=\ln 1+1-1=0.$ ∎

Lemma 13**.**

For positive integers $a>b$, and positive integer $j$,

[TABLE]

Proof:

That $\frac{f(b)}{f(a)}\leq 1$, where $f(x)\triangleq e^{-x}\left(1+\frac{x}{j}\right)^{x+j+\frac{1}{2}}$ follows directly from

[TABLE]

for all positive values of $x$. ∎

Lemma 14**.**

For positive integers $b,c$ such that $1\leq b<b+2\leq c,$

[TABLE]

Proof:

For $b=1$ the lemma follows because $c\geq b+2\geq 3$ and $c\leq 2^{c-1}$ for all $c\geq 2.$

For $b\geq 2,$

[TABLE]

implies that

[TABLE]

∎

Appendix B Lemma 4

Proof:

If

[TABLE]

then clearly

[TABLE]

by definition.

To prove Equation (5), first obtain a lower bound to the LHS of Equation (5) via

[TABLE]

where Equation (7) is an alternative equation for the variational distance. Next let $\tilde{q}\in\mathcal{P}(\mathcal{W}|\mathcal{X})$ be any distribution such that $q_{\hat{x}}(a)\geq i_{y}(a)$ for all $a\in\mathcal{W}.$ Now obtain an upper bound to Equation (5) via

[TABLE]

∎

Appendix C Proofs of main results

Proof:

To begin the proof, note the uniform average semantic loss for a given summarizer can be written

[TABLE]

where

[TABLE]

and $r$ is uniform over $\mathcal{P}(\mathcal{X})$, due to the integral function being linear. The proof proceeds by evaluating $\alpha(\mathcal{W},x^{n},y),$ and specifically showing

[TABLE]

To help in evaluating the integrals, assume that $\mathcal{X}=\{1,\dots,|\mathcal{X}|\}$, and let

[TABLE]

for all $m\in\mathcal{X}.$ Of importance throughout the proof will be that

[TABLE]

for all integers $m\in\mathcal{X},$ and that

[TABLE]

Two notable values are $\bar{t}_{1}=n+|\mathcal{X}|-1$ and $\bar{p}_{1}=1$. Also, without loss of generality assume that $\mathcal{W}\cap\mathcal{X}(y)=\{1,\dots,w\}$, $\mathcal{X}(y)=\{1,\dots,z\}$ and $x(1)=1.$

With this new notation

[TABLE]

where $\mathrm{d}p^{n}=\mathrm{d}p_{n}\mathrm{d}p_{n-1}\dots\mathrm{d}p_{1}$, since $r(p)=(|\mathcal{X}|-1)!$ by corollary 10 and $\mathcal{P}(\mathcal{X})$ is the convex set

[TABLE]

Of the two integrals in (16) we shall only show

[TABLE]

since

[TABLE]

follows similarly. At this point note that Equation (13) directly follows from Equations (16), (17) and (18), so validating Equation (18) would finish the proof. We shall prove Equation (17) through a rather tedious recursion process. To aid in this recursion we shall, in an abuse of notation, write $\mathcal{\tilde{P}}(k)$ to denote the convex set

[TABLE]

and use $\mathrm{d}p^{k}$ to denote the differential sequence $\mathrm{d}p_{k}\mathrm{d}p_{k-1}\dots\mathrm{d}p_{1}.$

Write the LHS of (17)

[TABLE]

by using $p_{|\mathcal{X}|}=\bar{p}_{|\mathcal{X}|}=\bar{p}_{|\mathcal{X}|-1}-p_{|\mathcal{X}|-1}$, via Equation (14). The inner integration can be performed via Lemma 9 yielding

[TABLE]

where (20) follows via Equation (14), this time to show $\bar{p}_{|\mathcal{X}|-1}=\bar{p}_{|\mathcal{X}|-2}-p_{|\mathcal{X}|-2}$ term. This process of using Lemma 9 to evaluate the integral, and then using Equation (14) to put the result into a form which can be evaluated using Lemma 9 can be repeated to evaluate the integrals over $p_{|\mathcal{X}|-2},\dots,p_{z+1}$; doing so yields

[TABLE]

At this point the recursion no longer directly applies since the next variable of integration, $p_{z}$, is contained in the denominator of the fraction. To address this, use the Taylor series expansion of $\frac{1}{1-x}$, specifically as follows

[TABLE]

Plugging (22) into (21) and exchanging the summations and integrals results in

[TABLE]

From here, evaluating all remaining integrals using the recursive process by which Equation (20) and (21) are obtained yields

[TABLE]

Then

[TABLE]

follows from “simplifying” (24). Equation (17) is therefore verified, completing the proof, having shown the LHS equals (25) since

[TABLE]

∎

Appendix D Proof of Lemma 7

Proof:

Let $s(j)\triangleq\frac{(b+j)!c!}{(c+j)!b!}$.

The lower bound follows primarily because

[TABLE]

Indeed given Equation (26)

[TABLE]

since $\frac{b}{c}<1$.

The upper bound is a bit more involved. Begin by letting $\phi$ and $\rho$ be arbitrary positive integers the values for which will be specified later. We are going to split the summation of $s(j)$ into three distinct regions, each of which will be given a different upper bound. This is done because $s(j)$ behaves differently depending on where it is in the summation terms, with earlier terms more resembling the Taylor series of $1/(1-x)$ while the later terms more resemble a geometric series. In specific,

[TABLE]

since $\frac{b+k}{c+k}=\frac{b}{c}+\frac{k}{c}\left(1-\frac{b+k}{c+k}\right)>\frac{b}{c}$ while

[TABLE]

follows by replacing all factorials with their appropriate counter parts from Robbin’s sharpening of Stirling’s formula444

$\sqrt{2\pi}n^{n+\frac{1}{2}}e^{-n+\frac{1}{12n+1}}\leq n!\leq\sqrt{2\pi}n^{n+\frac{1}{2}}e^{-n+\frac{1}{12n}}$

for all positive integers $n$. Now extending the first two summations to infinity, and making use of Lemmas 12 and 13 gives

[TABLE]

Since $\rho$ was arbitrary

[TABLE]

follows from Equations (30) by setting $\rho=2c$ and then recognizing the first two summations as Taylor series expansions of $1/(1-x)$, using [1, Lemma 10.5.3]555For $0\leq x,y\leq 1,~{}n>0,$

$(1-xy)^{n}\leq 1-x+e^{-yn}.$

to show

[TABLE]

and for the final summation using Lemmas 11, 14 and that $c\geq b+2>b\geq 1$.

Choosing $\phi=3c\frac{\ln(c-b)}{c-b}$ in Equation (31) yields

[TABLE]

∎

Appendix E Proof of Theorem 8

Proof:

First,

[TABLE]

by Lemma 7, implying that

[TABLE]

Likewise

[TABLE]

by Lemma 7, thus showing (which we will do momentarily)

[TABLE]

implies

[TABLE]

Combining Equation (34) and (37) proves the theorem.

Returning to prove Equation (36), we will need the three following technical results.

The first technical result is

[TABLE]

for all $\rho\in\mathcal{P}_{n}(\mathcal{X})$, $x^{n}$ such that $\pi^{(x^{n})}=\rho,$ and $a\in\mathcal{X}.$ Indeed, this is a straightforward combinatorial result since the term in the is simply $|\mathcal{T}^{n-1}_{(x(2),x(3),\dots,x(n))}|$, that is

[TABLE]

because fixing the empirical distribution of $n$-length sequences over $\mathcal{X}$, and fixing the first symbol in the sequence, also fixes the empirical distribution of the second through $n$-th symbols.

The second technical result is

[TABLE]

This technical result is a consequence of

[TABLE]

where the initial equality can be easily seen via a “stars and bars” proof, where the first bar must be chosen after the $\sqrt{n}$-th star.

The third and final technical result is

[TABLE]

for all $a\in\mathcal{X}.$ Indeed, this result follows from combining

[TABLE]

and

[TABLE]

where the first equality is due to (40), and the second equality due to the inner sum must being equal for all $a\in\mathcal{X}$ by symmetry .

Now with these technical results in tow, and recognizing that $\varepsilon(k)$ is monotonically decreasing with $k$ shows that the LHS of Equation (36) is less than or equal to

[TABLE]

Next splitting up the summation by the type sets and which report needs to be summarizer, the above bounding can be continued with

[TABLE]

where the fact that $|\mathcal{X}(y)|=|\mathcal{X}|2^{-j}$ for a length $j$-summarizer has been used. To which we may apply our first technical result, Equation (40), yielding

[TABLE]

Which itself is

[TABLE]

by $\varepsilon(k)$ being a monotonically increasing function of $k$ and recognizing that $n\rho(a)<\sqrt{n}$ also means that $\rho(a)\leq\frac{1}{\sqrt{n}}$. Finally applying our second and third technical results, Equations (41) and (44), yields

[TABLE]

∎