Minimax bounds for structured prediction

TL;DR

This paper establishes fundamental limits on the number of samples needed for learning structured prediction models, specifically factor-graph inference models, providing a theoretical understanding of their sample complexity.

Contribution

It introduces minimax bounds for factor-graph structured prediction models, revealing the minimal sample size required for any algorithm to successfully learn these predictors.

Findings

Derived minimax bounds for factor-graph models

Characterized the necessary sample complexity for structured prediction

Provided theoretical limits applicable to all algorithms

Abstract

Structured prediction can be considered as a generalization of many standard supervised learning tasks, and is usually thought as a simultaneous prediction of multiple labels. One standard approach is to maximize a score function on the space of labels, which decomposes as a sum of unary and pairwise potentials, each depending on one or two specific labels, respectively. For this approach, several learning and inference algorithms have been proposed over the years, ranging from exact to approximate methods while balancing the computational complexity. However, in contrast to binary and multiclass classification, results on the necessary number of samples for achieving learning is still limited, even for a specific family of predictors such as factor graphs. In this work, we provide minimax bounds for a class of factor-graph inference models for structured prediction. That is, we characterize the necessary sample complexity for any conceivable algorithm to achieve learning of factor-graph predictors.

Equations

$$f(x,y)=\sum _{\varphi \in \Phi }{f}_{\varphi }(x,y_{\varphi }).$$

$$R_P(f)=\underset{(x,y)\sim P}{E}[L_H(f(x),y)]\mathrm{and}{R}_S(f)=\frac{1}{m}\sum _{(x,y)\in S}{L}_H(f(x),y).$$

$${\mathbb{E}}_P[\rho (\hat{\theta }(z_1,\dots ,z_m),\theta (P))],$$

$$\underset{P\in \mathcal{P}}{\sup }{\mathbb{E}}_P[\rho (\hat{\theta }(z_1,\dots ,z_m),\theta (P))].$$

$${\mathfrak{M}}_m(\theta (\mathcal{P}),\rho ):=\underset{\hat{\theta }}{\inf }\underset{P\in \mathcal{P}}{\sup }{\mathbb{E}}_P\left[\rho (\hat{\theta }(z_1,\dots ,z_m),\theta (P))\right],$$

$${\mathfrak{M}}_m(\mathcal{P})=\underset{\mathcal{A}}{\inf }\underset{P\in \mathcal{P}}{\sup }\underset{S\sim P^m}{E}[R_P(\mathcal{A}(S))-R_P(f^{\ast })],$$

$$\text{max-}\{0,1{\}}^2\text{-Dim}(\mathcal{F}(G))=\underset{(u,v)\in T}{\max }\{0,1{\}}^2\text{-Dim}({\mathcal{F}}_{u,v}^{(0)}),$$

$${\mathfrak{M}}_m(\mathcal{P})\ge \frac{1}{81}\min \left(\frac{d-1}{\gamma m},\sqrt{\frac{d-1}{m}}\right).$$

$${\mathfrak{M}}_m(\mathcal{P})\ge \underset{(u,v)\in T}{\max }{\mathfrak{M}}_m({\mathcal{D}}_{\gamma ,u,v}).$$

$$\displaystyle\mathbb{P}_{\gamma,u,v,B}^{(x)}[x_{i}]=\left\{\begin{array}[]{ll}p,&\text{if }i\in\{1,\ldots,d^{(0)}_{u,v}-1\}\\ 1-(d^{(0)}_{u,v}-1)p,&\text{otherwise.}\end{array}\right.$$

$$\displaystyle\mathbb{P}_{\gamma,u,v,B}^{(y|x)}[y|x]=\left\{\begin{array}[]{ll}\frac{1-3\gamma}{4},&\text{if }x=x_{i},\ \mathrm{and}\ y_{u}=1-B_{i1},\ \mathrm{and}\ y_{v}=1-B_{i2},\\ &\mathrm{and}\ y_{k}=0\text{ for }k\in V\setminus\{u,v\},\ \mathrm{and}\ i\in\{1,\ldots,d^{(0)}_{u,v}-1\}\\ \frac{1+\gamma}{4},&\text{if }x=x_{i},\ \mathrm{and}\ y_{k}=0\text{ for }k\in V\setminus\{u,v\},\ \mathrm{and}\ i\in\{1,\ldots,d^{(0)}_{u,v}-1\}\\ 0,&\text{otherwise,}\end{array}\right.$$

$$\displaystyle\eta_{j}^{(\gamma,u,v,B)}(x)\equiv\mathbb{P}^{(y_{j}|x)}_{\gamma,u,v,B}[y_{j}=1|x]=\left\{\begin{array}[]{ll}{\frac{1-\gamma}{2},}&\text{if }x=x_{i}\text{ for some }i\in\{1,\ldots,d^{(0)}_{u,v}-1\},\mathrm{and}\\ &((j=u\ \mathrm{and}\ B_{i1}=0)\ \mathrm{or}\ (j=v\ \mathrm{and}\ B_{i2}=0))\\ {\frac{1+\gamma}{2},}&\text{if }x=x_{i}\text{ for some }i\in\{1,\ldots,d^{(0)}_{u,v}-1\},\mathrm{and}\\ &((j=u\ \mathrm{and}\ B_{i1}=1)\ \mathrm{or}\ (j=v\ \mathrm{and}\ B_{i2}=1))\\ {0,}&{\text{otherwise,}}\end{array}\right.$$

$$\displaystyle(f_{B,u,v}^{*}(x))_{j}=\left\{\begin{array}[]{ll}0,&\text{ if }x=x_{i}\text{ for some }i\in\{1,\ldots,d^{(0)}_{u,v}-1\},\\ &\mathrm{and}\ ((j=u\ \mathrm{and}\ B_{i1}=0)\ \mathrm{or}\ (j=v\ \mathrm{and}\ B_{i2}=0))\\ 1,&\text{ if }x=x_{i}\text{ for some }i\in\{1,\ldots,d^{(0)}_{u,v}-1\},\\ &\mathrm{and}\ ((j=u\ \mathrm{and}\ B_{i1}=1)\ \mathrm{or}\ (j=v\ \mathrm{and}\ B_{i2}=1))\\ 0,&\text{ otherwise.}\end{array}\right.$$

$$R_{B,u,v}(\mathcal{A}(S))-R_{B,u,v}(f_{B,u,v}^{\ast })=\underset{(x,y)\sim D_{\gamma ,u,v,B}}{E}\left[\sum _{j=1}^l\left(1-2y_j\right)\left(({\hat{f}}_m(x){)}_j-(f_{B,u,v}^{\ast }(x){)}_j\right)\right]$$

$$=\sum _{j=1}^l\underset{x\sim D_{\gamma ,u,v,B}^{(x)}}{E}\left[\underset{y_j\sim D_{\gamma ,u,v,B}^{(y_j|x)}}{E}\left[\left(1-2y_j\right)\left(({\hat{f}}_m(x){)}_j-(f_{B,u,v}^{\ast }(x){)}_j\right)\right]\right]$$

$$=\sum _{j=1}^l\underset{x\sim D_{\gamma ,u,v,B}^{(x)}}{E}\left[\left|2{\eta }_j^{(\gamma ,u,v,B)}(x)-1\right|\cdot \left|({\hat{f}}_m(x){)}_j-(f_{B,u,v}^{\ast }(x){)}_j\right|\right]$$

$$\ge \gamma \cdot \underset{x\sim D_{\gamma ,u,v,B}^{(x)}}{E}\left[\sum _{j=1}^l\left|({\hat{f}}_m(x){)}_j-(f_{B,u,v}^{\ast }(x){)}_j\right|\right]$$

$$=\gamma \cdot \sum _{i=1}^{d_{u,v}^{(0)}}\sum _{j=1}^l\left|({\hat{f}}_m(x_i){)}_j-(f_{B,u,v}^{\ast }(x_i){)}_j\right|\cdot {\mathbb{P}}_{\gamma ,u,v,B}^{(x)}[x_i]\stackrel{\mathrm{def}}{=}\gamma \cdot \Vert {\hat{f}}_m-f_{B,u,v}^{\ast }{\Vert }_{1,1},$$

$${\mathfrak{M}}_m({\mathcal{D}}_{\gamma ,u,v})$$

$$\ge \gamma \cdot \underset{{\hat{f}}_m}{\inf }\underset{B\in \{0,1{\}}^{(d_{u,v}^{(0)}-1)\times 2}}{\max }{\mathbb{E}}_{B,u,v}\left[\Vert {\hat{f}}_m-f_{B,u,v}^{\ast }{\Vert }_{1,1}\right],$$

$${\hat{B}}_m$$

$$\Vert f_{{\hat{B}}_m,u,v}^{\ast }-f_{B,u,v}^{\ast }{\Vert }_{1,1}$$

$${\mathfrak{M}}_m({\mathcal{D}}_{\gamma ,u,v})\ge \frac{\gamma }{2}\underset{{\hat{B}}_m}{\inf }\underset{B\in \{0,1{\}}^{(d_{u,v}^{(0)}-1)\times 2}}{\max }{\mathbb{E}}_{B,u,v}\left[\Vert f_{{\hat{B}}_m,u,v}^{\ast }-f_{B,u,v}^{\ast }{\Vert }_{1,1}\right],$$

$$\Vert f_{B,u,v}^{\ast }-f_{B^{\prime },u,v}^{\ast }{\Vert }_{1,1}$$

$$=p\cdot \sum _{i=1}^{d_{u,v}^{(0)}-1}\sum _{j=1}^2\left|B_{ij}-B_{ij}^{\prime }\right|=p\cdot L_H(B,B^{\prime }).$$

$${\mathfrak{M}}_m({\mathcal{D}}_{\gamma ,u,v})\ge \frac{p\gamma }{2}\underset{{\hat{B}}_m}{\inf }\underset{B\in \{0,1{\}}^{(d_{u,v}^{(0)}-1)\times 2}}{\max }{\mathbb{E}}_{B,u,v}\left[L_H({\hat{B}}_m,B)\right],$$

$$H^2(D_{\gamma ,u,v,B},D_{\gamma ,u,v,B^{\prime }})$$

$$=p\cdot \sum _{i=1}^{d_{u,v}^{(0)}-1}\sum _{y\in \{0,1{\}}^l}{\left(\sqrt{{\mathbb{P}}_{\gamma ,u,v,B^{\prime }}^{(y|x)}(y|x_i)}-\sqrt{{\mathbb{P}}_{\gamma ,u,v,B}^{(y|x)}(y|x_i)}\right)}^2.$$

$${\mathfrak{M}}_m({\mathcal{D}}_{\gamma ,u,v})\ge \frac{p\gamma }{2}\underset{{\hat{B}}_m}{\inf }\underset{B\in \{0,1{\}}^{(d_{u,v}^{(0)}-1)\times 2}}{\max }{\mathbb{E}}_{B,u,v}\left[L_H({\hat{B}}_m,B)\right]\ge \frac{p\gamma (d_{u,v}^{(0)}-1)}{2}(1-\sqrt{6p{\gamma }^{2}m})$$

$${\mathfrak{M}}_m({\mathcal{D}}_{\gamma ,u,v})\ge \frac{d_{u,v}^{(0)}-1}{81\gamma m}.$$

$${\mathfrak{M}}_m({\mathcal{D}}_{\gamma ,u,v})\ge \frac{d_{u,v}^{(0)}-1}{81\tilde{\gamma }m}=\frac{1}{81}\sqrt{\frac{d_{u,v}^{(0)}-1}{m}}.$$

$${\mathfrak{M}}_m(\mathcal{P})$$

$$=\frac{1}{81}\min \left(\frac{\text{max-}\{0,1{\}}^2\text{-Dim}(\mathcal{F})-1}{\gamma m},\sqrt{\frac{\text{max-}\{0,1{\}}^2\text{-Dim}(\mathcal{F})-1}{m}}\right).$$

$${\mathcal{H}}_{11}=\{h(\cdot )=g(\cdot {)}_{1}g(\cdot {)}_2\mid g\in \mathcal{G}\},$$

Full text

1 Introduction

Structured prediction has been continuously used over the years in multiple domains such as computer vision, natural language processing, and computational biology. Key examples of structured prediction problems include image segmentation, dependency parsing, part-of-speech tagging, named entity recognition, machine translation and protein folding. In this setting, the input $x$ is some observation, e.g., social network, an image, a sentence. The output is a labeling $y$, e.g., an assignment of each individual of a social network to a cluster, or an assignment of each pixel in the image to foreground or background, or an acyclic graph as in dependency parsing. A property common to these tasks is that, in each case, the natural loss function admits a decomposition along the output substructures. Thus, a common approach to structured prediction is to exploit local features to infer the global structure. For instance, one could include a feature that encourages two individuals of a social network to be assigned to different clusters whenever there is a strong disagreement in opinions about a particular subject. Then, one can define a posterior distribution over the set of possible labelings conditioned on the input.

The output structure and corresponding loss function make these problems significantly different from the (unstructured) binary or multiclass classification problems extensively studied in learning theory. Some classical algorithms for learning the parameters of the model include conditional random fields [14], structured support vector machines [22, 24, 2], kernel-regression algorithm [8], search-based structured prediction [10]. More recently, deep learning algorithms have been developed for specific tasks such as image annotation [27], part-of-speech-tagging [13, 26], and machine translation [30].

However, in contrast to the several algorithms developed, there have been relatively few studies devoted to the theoretical understanding of structured prediction. From the few theoretical literature, the most studied aspect has been the generalization error bounds. [6, 5, 23] provided learning guarantees that hold primarily for losses such as the Hamming loss and apply to specific factor graph models. [16, 12, 3, 11] provide PAC-Bayesian guarantees for arbitrary losses through the analysis of randomized algorithms using count-based hypotheses. Literature on lower bounding the sample complexity for structure prediction is scarcer even for specific hypothesis classes of losses. Information-theoretic bounds have been studied in the context of binary graphical models [18, 20] and Gaussian Markov random fields [29]. Nevertheless, there is still a lack of understanding in the context of more general structured prediction problems.

Our main contribution consists of characterizing the necessary sample complexity for learning factor graph models in the context of structured prediction. Specifically, in Theorem 1, we show that the finiteness of the $\{0,1\}^{2}$-dimension (see Definition 3) is necessary for learning. We further show in Theorem 2 the connection of the $\{0,1\}^{2}$-dimension to the VC-dimension [25], which will allow us to compute the $\{0,1\}^{2}$-dimension from the several known results on VC-dimension.

2 Preliminaries

Let ${\mathcal{X}}$ denote the input space and ${\mathcal{Y}}$ the output space. In structured prediction, the output space usually consists of a large (e.g., exponential) set of discrete objects admitting some possibly overlapping structure. Among common structures in the literature, one finds set of sequences, graphs, images, parse trees, etc. Thus, we consider the output space ${\mathcal{Y}}$ to be decomposable into $l$ substructures: ${\mathcal{Y}}={\mathcal{Y}}_{1}\times\dots\times{\mathcal{Y}}_{l}$. Here, ${\mathcal{Y}}_{i}$ is the set of possible labels that can be assigned to substructure $i$. For example, in a webpage collective classification task [21], each ${\mathcal{Y}}_{i}$ is a webpage label, whereas ${\mathcal{Y}}$ is a joint label for an entire website. In this work we assume that ${\mathcal{Y}}_{i}\in\{0,1\}$, that is, $|{\mathcal{Y}}_{i}|=2$ for all $i$. In this case, the number of possible assignments to ${\mathcal{Y}}$ is exponential in the number of substructures $l$, i.e., $|{\mathcal{Y}}|=2^{l}$.

The Hamming loss.

In order to measure the success of a prediction, we use the Hamming loss throughout this work. Specifically, for two outputs $y,y^{\prime}\in{\mathcal{Y}}$, with $y=(y_{1},\dots,y_{l})$ and $y^{\prime}=(y^{\prime}_{1},\dots,y^{\prime}_{l})$, the Hamming loss, $L_{H}$, is defined as $L_{H}(y,y^{\prime})=\sum_{i=1}^{l}\mathbbm{1}\left[y_{i}\neq y^{\prime}_{i}\right]$. The Hamming loss has been widely used in structured prediction, for instance, in image segmentation one may count the number of pixels that are incorrectly assigned as foreground/background; in graphs, one may count the number of different edges between the prediction and the true label.

Factor graphs and scoring functions.

We adopt a common approach in structured prediction where predictions are based on a scoring function mapping ${\mathcal{X}}\times{\mathcal{Y}}$ to ${\mathcal{R}}$. Let ${\mathcal{F}}$ be a family of scoring functions. For any $f\in{\mathcal{F}}$, we denote by $f(x)$ the predictor defined by $f$: for any $x\in{\mathcal{X}}$ , $f(x)=\operatorname*{arg\,max}_{y\in{\mathcal{Y}}}f(x,y).$

Furthermore, we assume that each function $f\in{\mathcal{F}}$ can be decomposed as a sum, as is standard in structured prediction. We consider the most general case for such decompositions through the notion of factor graphs, described also in [7]. A factor graph $G$ is a bipartite graph, and is represented as a tuple $G=(V,\Phi,E)$, where $V$ is a set of variable nodes, $\Phi$ a set of factor nodes, and $E$ a set of undirected edges between a variable node and a factor node. In our context, $V$ can be identified with the set of substructure indices, that is $V=\{1,\dots,l\}.$ We further assume that $G$ is connected. Note that, in contrast to graphical models, we do not assume $f$ to be a probabilistic model but it would also be captured by this framework.

For any factor node $\phi\in\Phi$, denote by $\mathrm{Scope}(\phi)\subseteq V$ the set of variable nodes connected to $\phi$ via an edge and define ${\mathcal{Y}}_{\phi}$ as the substructure set cross-product ${\mathcal{Y}}_{\phi}=\bigtimes_{i\in\mathrm{Scope}(\phi)}{\mathcal{Y}}_{i}$. Then, $f$ decomposes as a sum of functions $f_{\phi}$ , each taking as argument an element of the input space $x\in{\mathcal{X}}$ and an element of ${\mathcal{Y}}_{\phi}$, $y_{\phi}\in{\mathcal{Y}}_{\phi}$:

[TABLE]

Specifically, we focus on factor graphs with unary and pairwise factors, that is, each factor node $\phi\in\Phi$ is connected to one or two nodes in $V$. We let $\phi_{uv}$ denote a pairwise factor node connected to $u,v\in{\mathcal{V}}$, i.e., $\mathrm{Scope}(\phi_{uv})=\{u,v\}$. Then, the score induced from $\phi_{uv}$ is given by $f_{\phi_{uv}}(x,y_{\phi_{uv}})=f_{\phi_{uv}}(x,y_{u},y_{v})$. Note that in this case $\phi_{uv}$ and $\phi_{vu}$ represent the same factor node and induce the same score. Similarly, for unary factor nodes, we let $\phi_{u}$ denote a factor node connected to $u\in{\mathcal{V}}$ with score given by $f_{\phi_{u}}(x,y_{\phi_{u}})=f_{\phi_{u}}(x,y_{u})$. We further use ${\mathcal{F}}(G)$ to denote functions that are decomposable with respect to the graph $G$. Note also that while all $f\in{\mathcal{F}}(G)$ decompose with respect to same graph $G$, the score functions $f_{\phi}$ and $f^{\prime}_{\phi}$ are allowed to be different for any $\phi\in\Phi$, $f,f^{\prime}\in{\mathcal{F}}(G)$. Figure 1 shows different examples of factor graphs with unary and pairwise factors.

Learning.

We receive a training set $S=((x_{1},y_{1}),\dots,(x_{m},y_{m}))$ of $m$ i.i.d. samples drawn according to some distribution $P$ over ${\mathcal{X}}\times{\mathcal{Y}}$. We denote by $R_{P}(f)$ the expected Hamming loss and by $R_{S}(f)$ the empirical Hamming loss of $f$:

[TABLE]

Our learning scenario consists of using the sample $S$ to select a hypothesis $f\in{\mathcal{F}}(G)$ with small expected Hamming loss $R_{P}(f)$.

Next, we introduce the definition of Bayes-Hamming loss, which in words is the minimum attainable expected Hamming loss by any predictor.

Definition 1** (Bayes-Hamming loss).**

For any given distribution $P$ over ${\mathcal{X}}\times{\mathcal{Y}}$, the Bayes-Hamming loss is defined as the minimum achievable expected Hamming loss among all possible predictors $f\mathrel{\mathop{\ordinarycolon}}{\mathcal{X}}\to{\mathcal{Y}}$. That is, $R^{*}=\min_{f}R_{P}(f).$

Then the Bayes-Hamming predictor, $f^{*}$, is defined as the function that achieves the Bayes-Hamming loss, that is, $R(f^{*})=R^{*}$.

The following proposition shows how the Bayes-Hamming predictor makes its decision with respect to the Hamming loss.

Proposition 1**.**

For any given distribution $P$ over ${\mathcal{X}}\times{\mathcal{Y}}$, the Bayes-Hamming predictor $f^{*}$ is: $(f^{*}(x))_{i}=\mathbbm{1}\left[\eta_{i}(x)\geq 1/2\right].$ where $\eta_{i}(x)$ is the marginal probability $\operatorname*{\mathbb{P}}[y_{i}=1|x]$, for each substructure $y_{i}$.

(See Appendix A for detailed proofs.)

We emphasize that the above definition considers the Hamming loss, $L_{H}$, as defined at the beginning of Section 2. For other types of loss functions, the Bayes predictor can have different optimal decisions.

2.1 Minimax risk framework

The standard minimax risk consists of a family of distributions ${\mathcal{P}}$ over a sample space ${\mathcal{Z}}$, and a function $\theta\mathrel{\mathop{\ordinarycolon}}{\mathcal{P}}\to\Theta$ defined on ${\mathcal{P}}$, that is, a mapping $P\mapsto\theta(P)$. We aim to estimate the parameter $\theta(P)$ based on a sequence of i.i.d. observations $(z_{i})_{i=1}^{m}$ drawn from the (unknown) distribution $P$. To evaluate the quality of an estimator $\theta$, we let $\rho\mathrel{\mathop{\ordinarycolon}}\Theta\times\Theta\to{\mathbb{R}}_{+}$ denote a semi-metric on the space $\Theta$, which we use to measure the error of an estimator $\widehat{\theta}$ with respect to the parameter $\theta$. For a distribution $P\in{\mathcal{P}}$ and for a given estimator $\widehat{\theta}\mathrel{\mathop{\ordinarycolon}}{\mathcal{Z}}^{m}\to\Theta$, we assess the quality of the estimate $\widehat{\theta}(z_{1},\ldots,z_{m})$ in terms of the (expected) risk:

[TABLE]

where $\mathbb{E}_{P}[\cdot]$ denotes the expectation with respect to $(z_{1},\dots,z_{m})\sim P^{m}$. A common approach, first suggested by [28], for choosing an estimator $\widehat{\theta}$ is to select the one that minimizes the maximum risk, that is,

[TABLE]

An optimal estimator for this metric then gives the minimax risk, which is defined as:

[TABLE]

where we take the supremum (worst-case) over distributions $P\in{\mathcal{P}}$, and the infimum is taken over all estimators $\widehat{\theta}$. Here the notation $\theta({\mathcal{P}})$ indicates that we consider distributions in ${\mathcal{P}}$ and parameters $\theta(P)$ for $P\in{\mathcal{P}}$.

2.2 Minimax risk in structured prediction

We now apply the framework above to our context and study a specialized notion of risk appropriate for prediction problems. In this setting, we aim to estimate a function $f\in{\mathcal{F}}(G)$ by using samples from a distribution $P$. For any sample $(x,y)\sim P$, we will measure the quality of our estimation, $f$, by comparing its output $f(x)$ to the structure $y$ drawn from $P$ through the Hamming loss. By taking expectation, we obtain the expected risk or expected Hamming loss, $R_{P}(f)$ defined in eq.(1). We then compare this risk to the best possible Hamming loss, i.e., the Bayes-Hamming loss. That is, we assume that at least one function $f\in{\mathcal{F}}(G)$ achieves the Bayes-Hamming loss. Thus, we arrive to the following minimax excess risk:

[TABLE]

where $f^{*}=\operatorname*{arg\,min}_{f\in{\mathcal{F}}(G)}R_{P}(f)$, and ${\mathcal{A}}\mathrel{\mathop{\ordinarycolon}}({\mathcal{X}}\times{\mathcal{Y}})^{m}\to{\mathcal{F}}(G)$ is any algorithm that returns a predictor given $m$ training samples from $P$. Moreover, ${\mathcal{P}}$ defines a family of distributions over ${\mathcal{X}}\times{\mathcal{Y}}$. Intuitively speaking, for a fixed distribution $P\in{\mathcal{P}}$, the quantity $\mathfrak{M}_{m}({\mathcal{P}})$ represents the minimum expected excess loss achievable by any algorithm with respect to the factor graph $G$. Then $\mathfrak{M}_{m}({\mathcal{P}})$ looks into the distribution that attains the worst expected excess loss.

3 Information-theoretic lower bound for structured prediction

We are interested on finding a lower bound to the minimax risk (2) presented in Section 2.2. By doing this, we characterize the necessary number of samples to have any hope in achieving learning.

Before presenting our main result, we introduce a new type of dimension that will show up in our lower bound and will help to characterize learnability. Note that it is known that different notions of dimension of function classes help to characterize learnability in certain prediction problems. For example, for binary classification, the finiteness of the VC dimension [25] is necessary for learning [15]. For multiclass classification, it was shown that the finiteness of the Natarajan dimension is necessary for learning [9]. General notion of dimensions for multiclass classification has also been study in [4].

For a given function class ${\mathcal{G}}\subseteq\{g\ |\ g\mathrel{\mathop{\ordinarycolon}}{\mathcal{X}}\to\{0,1\}^{2}\}$, and dataset $S$ of $m$ samples, we use the following shorthand notation: ${\mathcal{G}}(S)=\{(g(x_{1}),\ldots,g(x_{m}))\in\{0,1\}^{m\times 2}\ |\ g\in{\mathcal{G}}\}.$ That is, ${\mathcal{G}}(S)$ contains all the matrices in $\{0,1\}^{m\times 2}$ that can be produced by applying all functions in ${\mathcal{G}}$ to the dataset $S$. Next we define the standard notion of shattering.

Definition 2** ($\{0,1\}^{2}$-shattering).**

A function class, ${\mathcal{G}}$, $\{0,1\}^{2}$-shatters a finite set $S$ of $m$ samples if ${\mathcal{G}}(S)$ produces all possible binary matrices in $\{0,1\}^{m\times 2}$. That is, $|{\mathcal{G}}(S)|=2^{2m}$.

Definition 3** ($\{0,1\}^{2}$-dimension).**

The $\{0,1\}^{2}$-dimension of a function class ${\mathcal{G}}$, denoted $\{0,1\}^{2}\text{-Dim}({\mathcal{G}})$, is the maximal size of a set $S$ that can be shattered by ${\mathcal{G}}$. If ${\mathcal{G}}$ can shatter sets of arbitrarily large size we say that ${\mathcal{G}}$ has infinite $\{0,1\}^{2}$-dimension.

The above dimension applies to functions with output in $\{0,1\}^{2}$. We will create functions with output in $\{0,1\}^{2}$ as follows. Let $f^{(0)}_{u,v}(x,y_{u},y_{v})=f\left(x,\left(0,\ldots,0,y_{u},0,\ldots,0,y_{v},0,\ldots,0\right)\right)$ denote the function $f(x,y)$ with $y_{i}=0$ for all $i\in\{1,\ldots,l\}\setminus\{u,v\}$. Then, let $f^{(0)}_{u,v}(x)=\operatorname*{arg\,max}_{y_{u},y_{v}}f^{(0)}_{u,v}(x,y_{u},y_{v})$, that is, the output of $f^{(0)}_{u,v}(x)$ is in $\{0,1\}^{2}$. The following dimension applies to function classes based on factor graphs.

Definition 4** ($\text{max-}\{0,1\}^{2}$-dimension).**

For a given factor graph $G=(V,\Phi,E)$, the $\text{max-}\{0,1\}^{2}$-dimension of a function class ${\mathcal{F}}(G)$, denoted as $\text{max-}\{0,1\}^{2}\text{-Dim}({\mathcal{F}}(G))$, is defined as:

[TABLE]

where $T=\{(u,v)\in\mathrm{Scope}(\phi)\ |\ \forall\phi\in\Phi\}$, and ${\mathcal{F}}^{(0)}_{u,v}=\{f^{(0)}_{u,v}\ |\ f\in{\mathcal{F}}(G),(u,v)\in T\}$.

Theorem 1**.**

Let $G=(V,\Phi,E)$ be a factor graph with pairwise and unary factors, let ${\mathcal{F}}(G)$ denote a class of functions $f\mathrel{\mathop{\ordinarycolon}}{\mathcal{X}}\to\{0,1\}^{l}$, where each $f\in{\mathcal{F}}(G)$ decomposes according to $G$, and let $d=\text{max-}\{0,1\}^{2}\text{-Dim}({\mathcal{F}}(G))\geq 2$. Then, we have that for any $\gamma\in[0,\nicefrac{{1}}{{3}}]$ and any $m\geq d$:

[TABLE]

Proof.

The proof is motivated by the work of [15] for binary classifiers. As a first step it is clear that one can lower bound eq.(2) by defining the maximum over a subset of ${\mathcal{P}}$. That is, we create a collection of family of distributions ${\mathbb{D}}_{\gamma}$, where $|{\mathbb{D}}|=|\Phi|$. Each family distribution ${\mathcal{D}}_{\gamma,u,v}\in{\mathbb{D}}_{\gamma}$ is further indexed by $(u,v)\in T=\{(u,v)\in\mathrm{Scope}(\phi)\ |\ \forall\phi\in\Phi\}$. Then we have,

[TABLE]

Our approach consists of first defining the families of distributions ${\mathcal{D}}_{\gamma,u,v}\subset{\mathcal{P}}$ such that its elements can be naturally indexed by the vertices of a binary hypercube. We will then relate the expected excess risk problem to an estimation of binary strings in order to apply Assouad’s lemma.

Construction of ${\mathcal{D}}_{\gamma,u,v}$.

Consider a fixed $(u,v)\in T$. We first focus on constructing a family of distributions, ${\mathcal{D}}_{\gamma,u,v}$, parameterized by $\gamma>0$. Each distribution $D_{\gamma,u,v,B}\in{\mathcal{D}}_{\gamma,u,v}$ is further indexed by a binary matrix $B\in\{0,1\}^{(d^{(0)}_{u,v}-1)\times 2}$, where $d^{(0)}_{u,v}$ is the $\{0,1\}^{2}$-dimension of ${\mathcal{F}}^{(0)}_{u,v}$. To construct these distributions, we will first pick the marginal distribution $D^{(x)}_{\gamma,u,v,B}$ of the feature $x$, and then specify the conditional distributions $D^{(y|x)}_{\gamma,u,v,B}$ of $y$ given $x$, for each $B\in\{0,1\}^{(d^{(0)}_{u,v}-1)\times 2}$.

We construct $D_{\gamma,u,v,B}^{(x)}$ as follows. Since ${\mathcal{F}}^{(0)}_{u,v}$ is a class with $\{0,1\}^{2}$-dimension $d^{(0)}_{u,v}$, there exists a set of points $\{x_{1},\ldots,x_{d^{(0)}_{u,v}}\}\in{\mathcal{X}}$ that are shattered by ${\mathcal{F}}^{(0)}_{u,v}$, that is, for any binary matrix $B\in\{0,1\}^{d^{(0)}_{u,v}\times 2}$ there exists at least one function $f^{(0)}_{u,v}\in{\mathcal{F}}^{(0)}_{u,v}$ such that $f^{(0)}_{u,v}(x_{i})=B_{i*}$, for all $i\in\{1,\ldots,d^{(0)}_{u,v}\}$. We now define the marginal distribution $D_{\gamma,u,v,B}^{(x)}$ such that its support is the shattered set $\{x_{1},\ldots,x_{d^{(0)}_{u,v}}\}$, i.e., $\operatorname*{\mathbb{P}}_{\gamma,u,v,B}^{(x)}[\{x_{1},\ldots,x_{d^{(0)}_{u,v}}\}]=1$. For a given parameter $p\in[0,\nicefrac{{1}}{{(d^{(0)}_{u,v}-1)}}]$, whose value is set later, we have:

[TABLE]

Next, for a fixed $B\in\{0,1\}^{(d^{(0)}_{u,v}-1)\times 2}$, the conditional distribution of $y$ given $x$, $D_{\gamma,u,v,B}^{(y|x)}$, is defined as:

[TABLE]

here we implicitly assume that $\gamma\in(0,1/3]$ in order to obtain a valid distribution. The above definition produces the following marginal probabilities:

[TABLE]

where we note that for each $j\in V$ and any $x$ we have that $|2\eta_{j}^{(\gamma,u,v,B)}(x)-1|\geq\gamma$. Given the above marginals, the corresponding Bayes-Hamming predictor for substructure $y_{j}$ for a given input $x$ (see Proposition 1), which we denote by $(f^{*}_{B,u,v}(x))_{j}$, is given by:

[TABLE]

That is, we have that the output of the Bayes-Hamming predictor on each $x_{i}$ for $i\in\{1\ldots d^{(0)}_{u,v}-1\}$, for each substructure $y_{j}$ for $j\in\{u,v\}$, is equal to the bit value $B_{i1}$ or $B_{i2}$, and zero otherwise.

Reduction to estimation of binary strings.

For any distribution $D_{\gamma,u,v,B}\in{\mathcal{D}}_{\gamma,u,v}$, we can further express the expected excess risk in eq.(2) as follows:

[TABLE]

where $R_{B,u,v}$ denotes the expected risk and $f^{*}_{B,u,v}$ the Bayes-Hamming predictor, both with respect to $D_{\gamma,u,v,B}$. Here $\hat{f}_{m}$ is the output of ${\mathcal{A}}(S)$, with $(\hat{f}_{m}(x))_{j}$ denoting the $j$-th substructure of the output $\hat{f}_{m}(x)$, and $\eta^{(\gamma,u,v,B)}_{j}(x)$ denotes the marginal probability $\mathbb{P}_{D^{(y_{j}|x)}_{\gamma,u,v,B}}[y_{j}=1|x]$. Equation (15) follows from our definition of $D^{(y_{j}|x)}_{\gamma,u,v,B}$ (see eq.(8)), and the $L_{1,1}$ matrix norm in eq.(16) is computed with respect to $D_{\gamma,u,v,B}^{(x)}$. Thus, we have that:

[TABLE]

where $\mathbb{E}_{B,u,v}[\cdot]$ denotes the expectation with respect to $S\sim D^{m}_{\gamma,u,v,B}$. Equation (17) follows from eq.(16). Given any candidate estimation $\hat{f}_{m}$, let $\widehat{B}_{m}\in\{0,1\}^{(d^{(0)}_{u,v}-1)\times 2}$ be defined as follows:

[TABLE]

Intuitively, $\widehat{B}_{m}$ is the binary matrix that indexes the element of $\{f^{*}_{B,u,v}\mathrel{\mathop{\ordinarycolon}}B\in\{0,1\}^{(d^{(0)}_{u,v}-1)\times 2}\}$ which is the closest to $\hat{f}_{m}$ in $L_{1,1}$ norm. Then, for any $B$, we have

[TABLE]

where we first applied the triangle inequality, and then used eq.(18). Applying this to eq.(17), we obtain:

[TABLE]

here the infimum is over all estimators that take values in $\{0,1\}^{(d^{(0)}_{u,v}-1)\times 2}$ based on $m$ samples, i.e., over $\widehat{B}_{m}\mathrel{\mathop{\ordinarycolon}}({\mathcal{X}}\times{\mathcal{Y}})^{m}\to\{0,1\}^{(d^{(0)}_{u,v}-1)\times 2}$. We now compute $\lVert f^{*}_{B,u,v}-f^{*}_{B^{\prime},u,v}\rVert_{1,1}$ for any two $B,B^{\prime}$. Using eq.(14) we have:

[TABLE]

In the last equality we abuse notation and consider the matrix $B\in\{0,1\}^{(d^{(0)}_{u,v}-1)\times 2}$ as a vector of dimension $2(d^{(0)}_{u,v}-1)$. Replacing this result into eq.(19), we get:

[TABLE]

which is related to an estimation problem in the $\{0,1\}^{2(d^{(0)}_{u,v}-1)}$ hypercube.

Applying Assouad’s lemma.

In order to apply Assouad’s lemma, we need an upper bound on the squared Hellinger distance $H^{2}(D_{\gamma,u,v,B},D_{\gamma,u,v,B^{\prime}})$ for all $B,B^{\prime}$ with $L_{H}(B,B^{\prime})=1$. For any two $B,B^{\prime}\in\{0,1\}^{(d^{(0)}_{u,v}-1)\times 2}$ we have:

[TABLE]

In the above summation, the inner sum is zero if $B_{i*}=B^{\prime}_{i*}$. Since we are interested on $B$ and $B^{\prime}$ such that $L_{H}(B,B^{\prime})=1$, this implies that for only one row $i$ from $\{1,\ldots,d^{(0)}_{u,v}-1\}$ we have $B_{i*}\neq B^{\prime}_{i*}$ with exactly one bit different. Then, the Hellinger distance results in: $H^{2}(D_{\gamma,u,v,B},D_{\gamma,u,v,B^{\prime}})=p\cdot(1-\gamma-\sqrt{1-2\gamma-3\gamma^{2}})\leq 6p\gamma^{2}.$ Applying Assouad’s lemma we obtain:

[TABLE]

Let $p=\nicefrac{{2}}{{(27\gamma^{2}m)}}$, and noting that if $\gamma\geq\sqrt{(d^{(0)}_{u,v}-1)/m}$ then the condition $p\leq\nicefrac{{1}}{{(d^{(0)}_{u,v}-1)}}$ holds. Replacing $p$ in eq.(20) we have:

[TABLE]

If $\gamma\leq\sqrt{(d^{(0)}_{u,v}-1)/m}$, and using the same construction as above with $\widetilde{\gamma}=\sqrt{(d^{(0)}_{u,v}-1)/m}$, we see that:

[TABLE]

Therefore, combining equations (21) and (22), and since the choice of $(u,v)$ was arbitrary, we have that:

[TABLE]

∎

4 Relation of $\{0,1\}^{2}$-dimension to VC-dimension

In this section, we show a connection of our defined $\{0,1\}^{2}$-dimension to the classical VC-dimension [25].

The following theorem shows that for a function class ${\mathcal{G}}\mathrel{\mathop{\ordinarycolon}}{\mathcal{X}}\to\{0,1\}^{2}$, the $\{0,1\}^{2}$-dimension of ${\mathcal{G}}$ is related to the minimum VC-dimension of a subclass of functions derived from ${\mathcal{G}}$.

Theorem 2**.**

Let ${\mathcal{G}}\subseteq\{g\ |\ g\mathrel{\mathop{\ordinarycolon}}{\mathcal{X}}\to\{0,1\}^{2}\}$ be a function class. Let ${\mathcal{H}}_{11},{\mathcal{H}}_{10},{\mathcal{H}}_{01},{\mathcal{H}}_{00}\subseteq{\{h\ |\ h\mathrel{\mathop{\ordinarycolon}}{\mathcal{X}}\to\{0,1\}\}}$ be four function classes defined as

[TABLE]

We have that $\{0,1\}^{2}\text{-Dim}({\mathcal{G}})=\min(\text{VC-Dim}({\mathcal{H}}_{11}),\text{VC-Dim}({\mathcal{H}}_{10}),\text{VC-Dim}({\mathcal{H}}_{01}),\text{VC-Dim}({\mathcal{H}}_{00}))$.

5 Discussion

We consider the problem of finding the necessary number of samples for learning of factor graphs with unary a pairwise factors. Our work was based on the minimax framework, that is, in obtaining a lower bound to the minimax risk. We showed a lower bound that requires the $\text{max-}\{0,1\}^{2}$-dimension to be finite in order for a function class to be learnable. We also note that in the proof of Theorem 1, our choice of setting a value of zero to many $y$’s was for clarity purposes. In principle, one can create such distributions by fixing $y$’s to arbitrary values in $\{0,1\}^{l-2}$, and this would result in a slightly different notion of dimension, which would take the maximum across the $2^{l-2}$ different values. However, our focus was on providing a clear guideline to obtain lower bounds in structured prediction, hence, we opted for simplicity. In addition, in Theorem 2 we showed the connection of the $\{0,1\}^{2}$-dimension to the VC-dimension, for which there are several known results for different types of function classes.

An interesting future work is the analysis of tightness. For example, regarding tightness for linear classifiers, consider inputs $x\in{\mathcal{R}}^{k}$. We observe that our lower bound in Theorem 1 is tight with respect to $k$ and $m$. Specifically, consider non-sparse linear classifiers as unary and pairwise potentials, Theorem 2 in [7] gives ${\mathcal{O}}(\sqrt{\nicefrac{{k}}{{m}}})$. In this case, the $\{0,1\}^{2}$-dimension is equal to the VC-dimension, and the latter is equal to $k$. Thus, we obtain a lower bound with rate $\sqrt{\nicefrac{{k}}{{m}}}$ for some $\gamma$. Similarly, consider sparse linear classifiers as unary and pairwise potentials. Then, Theorem 2 of [7] gives ${\mathcal{O}}(\sqrt{\nicefrac{{\log k}}{{m}}})$. In this case, the VC-dimension is ${\mathcal{O}}(\log k)$ [17], thus, we obtain a lower bound with rate $\sqrt{\nicefrac{{\log k}}{{m}}}$ for some $\gamma$. However, it remains to analyze for general functions where one possible attempt is perhaps to find an upper bound to the factor graph Rademacher complexity [7] in terms of the $\{0,1\}^{2}$-dimension, similar in spirit to the known result of the VC-dimension being an upper bound of the classical Rademacher complexity (see for instance, [19]).

Appendix A Detailed Proofs

A.1 Proof of Proposition 1

Proof.

Recall that $\eta_{i}(x)=\operatorname*{\mathbb{P}}[y_{i}=1|x]$. From eq.(1) and Definition 1, the Bayes-Hamming predictor $f^{*}$ minimizes the following expression (with respect to $f$)

[TABLE]

In order to minimize the above expression, for any $x$ we choose $(f(x))_{i}=1$ if $\eta_{i}(x)\geq 1/2$, and $(f(x))_{i}=0$ otherwise. ∎

A.2 Proof of Theorem 2

Proof.

Recall that for a dataset $S$ of $m$ samples, ${\mathcal{G}}(S)=\{(g(x_{1}),\ldots,g(x_{m}))\in\{0,1\}^{m\times 2}\ |\ g\in{\mathcal{G}}\}$. Similarly, define ${\mathcal{H}}_{ij}(S)=\{(h(x_{1}),\ldots,h(x_{m}))\in\{0,1\}^{m}\ |\ h\in{\mathcal{H}}_{ij}\}$ for all $i,j\in\{0,1\}$. Let $\{0,1\}^{2}\text{-Dim}({\mathcal{G}})=d$.

There exists a dataset $S$ of $d$ samples such that $|{\mathcal{G}}(S)|=2^{2d}$. Thus for all $i,j\in\{0,1\}$ we have $|{\mathcal{H}}_{ij}(S)|=2^{d}$, which implies that for all $i,j\in\{0,1\}$ we have $\text{VC-Dim}({\mathcal{H}}_{ij})\geq d$. Therefore $\min(\text{VC-Dim}({\mathcal{H}}_{11}),\text{VC-Dim}({\mathcal{H}}_{10}),\text{VC-Dim}({\mathcal{H}}_{01}),\text{VC-Dim}({\mathcal{H}}_{00}))\geq d$.

Also, for any dataset $S$ of $d+1$ samples we have $|{\mathcal{G}}(S)|<2^{2(d+1)}$. Thus there exists $i,j\in\{0,1\}$ such that $|{\mathcal{H}}_{ij}(S)|<2^{d+1}$, implying that there exists $i,j\in\{0,1\}$ such that $\text{VC-Dim}({\mathcal{H}}_{ij})<d+1$. Therefore $\min(\text{VC-Dim}({\mathcal{H}}_{11}),\text{VC-Dim}({\mathcal{H}}_{10}),\text{VC-Dim}({\mathcal{H}}_{01}),\text{VC-Dim}({\mathcal{H}}_{00}))<d+1$.

From the above, $\min(\text{VC-Dim}({\mathcal{H}}_{11}),\text{VC-Dim}({\mathcal{H}}_{10}),\text{VC-Dim}({\mathcal{H}}_{01}),\text{VC-Dim}({\mathcal{H}}_{00}))=d$. ∎