Epsilon-saturation for stable graphs and Littlestone classes

TL;DR

This paper introduces epsilon-saturation for stable graphs and Littlestone classes, exploring their properties, closure, and the relationship between Littlestone/stability and VC dimension, revealing surprising phenomena.

Contribution

It defines and characterizes epsilon-saturation, showing how it preserves stability and Littlestone properties under certain conditions, and analyzes its impact on complexity measures.

Findings

Epsilon-saturation forms a closure of the class under virtual elements.

Under certain parameters, epsilon-saturation remains Littlestone or stable.

The relationship between Littlestone/stability and VC dimension exhibits surprising phenomena.

Abstract

Any Littlestone class, or stable graph, has finite sets which function as ``virtual elements'': these can be seen from the learning side as representing hypotheses which are expressible as weighted majority opinions of hypotheses in the class, and from the model-theoretic side as an approximate finitary version of realizing types. We introduce and study the epsilon-saturation of a Littlestone class, or stable graph, which is essentially the closure of the class under inductively adding all such virtual elements. We characterize this closure and prove that under reasonable choices of parameters, it remains Littlestone (or stable), though not always of the same Littlestone dimension. This highlights some surprising phenomena having to do with regimes of epsilon and the relation between Littlestone/stability and VC dimension.

Equations

$$B\mbox{ is $\epsilon$-good }$$

$$A\mbox{ is $(\epsilon,\delta)$-excellent }$$

$$A\mbox{ is $\epsilon$-excellent.}$$

$$\{a_1,\dots ,a_k\}$$

$$H=\{(h_i,{\gamma }_i):i<k\}$$

$$\sum \{{\gamma }_i:h_i(x)=\mathbf{t}\}\ge 1-ϵ\text{mbox}andhence\sum \{{\gamma }_i:h_i(x)=1-\mathbf{t}\}<ϵ.$$

$$\{\mathbf{t}_{H}:H\mbox{ is an $\epsilon$-hypothesis of $\mathcal{H}$ }\}.$$

$$\langle x_{\eta}:\eta\in{{}^{s>}2}\rangle\mbox{ from X and }\langle y_{\rho}:\rho\in{{}^{s}2}\rangle\mbox{ from $\mathcal{H}_{\infty}$. }$$

$$E_{\rho}=\bigcup\{(h_{i},\gamma_{i})\in H_{\rho}~:~\mathbf{t}(h_{i},x_{\eta})\neq\mathbf{t}(H_{\rho},x_{\eta})\mbox{ for some $\eta$ with }\eta\triangleleft\rho\}.$$

$$E_{\eta}=\{(h_{i},\gamma_{i})\in H_{\eta}~:~\mathbf{t}(h_{i},x_{\rho})\neq\mathbf{t}(H_{\eta},x_{\rho})\mbox{ for some $\rho$ with }\eta\triangleleft\rho\}.$$

$$\mathrm{Ldim}(\mathcal{H})\ge s⟺\mathrm{Ldim}({\mathcal{H}}_{\infty }(ϵ))\ge s$$

$$\mathrm{VC}(\mathcal{H})\ge s⟺\mathrm{VC}({\mathcal{H}}_{\infty }(ϵ))\ge s.$$

$$\mathrm{Ldim}(\mathrm{dual}(\mathcal{H}))\ge s⟺\mathrm{Ldim}(\mathrm{dual}({\mathcal{H}}_{\infty }(ϵ)))\ge s$$

$$\mathrm{VC}(\mathrm{dual}(\mathcal{H}))\ge s⟺\mathrm{VC}(\mathrm{dual}({\mathcal{H}}_{\infty }(ϵ)))\ge s.$$

$$\mathrm{Ldim}(\mathcal{H})=\mathrm{Ldim}({\mathcal{H}}_{\infty }(ϵ)).$$

$$\mathrm{VC}(\mathcal{H})=\mathrm{VC}({\mathcal{H}}_{\infty }(ϵ)).$$

$$\sum \{w_t:h(x_t)\ne g(x_t),t<s\}<{ϵ}_L.$$

$$\sum \{{\gamma }_i:h_i(x)=g(x)\}\ge 1-{ϵ}_M.$$

$$P{r}_{x\sim \mathcal{D}}[h(x)\ne f(x)]<{ϵ}_1.$$

$$\forall h_f\in {\mathcal{H}}_f,P{r}_{x\sim \mathcal{D}}[h_f(x)=1]\ge {ϵ}_1\Rightarrow \exists s\in \mathcal{S}:h_f(s)=1.$$

$$\forall h\in \mathcal{H},P{r}_{x\sim \mathcal{D}}[h(x)\ne f(x)]\ge {ϵ}_1\Rightarrow \exists s\in \mathcal{S}:h(s)\ne f(s).$$

$$\sum \{{\gamma }_i:h_i(x)=1-\mathbf{t}(x),i<n\}<{ϵ}_2;$$

$${\mathrm{Maj}}_p(\mathcal{H})$$

$$\left|\left(\frac{|\{t<p:g_t(x)=1\}|}{|p|}\right)-\left(\sum \{{\gamma }_i:i<n,h_i(x)=1\}\right)\right|<{ϵ}_3.$$

$$\mathrm{Ldim}(\text{Maj}(\mathcal{H},\dots ,\mathcal{H}))\le \tilde{O}(p^2\cdot \mathrm{Ldim}(\mathcal{H})).$$

$${\mathcal{H}}_{\infty }(ϵ)\subseteq {\mathcal{H}}^{Cd-real}\subseteq {\mathrm{Maj}}_p(\mathcal{H})$$

$$A=\{(v_i,{\gamma }_i):i<k\}$$

$$\sum \{{\gamma }_i:v_i\sim v\}\ge 1-ϵ\text{mbox}or\sum \{{\gamma }_i:v_i\nsim v\}\ge 1-ϵ.$$

$$B=\{(v_i,{\gamma }_i):i<k\}$$

$$\sum \{{\gamma }_i:\mathbf{t}(A,v_i)=1\}\ge 1-ϵ\text{mbox}or\sum \{{\gamma }_i:\mathbf{t}(A,v_i)=0\}\ge 1-ϵ.$$

Full text

MM: Research partially supported by NSF-BSF 2051825.

OMMC: Research partially supported by Fundación Alberto y Dolores de Andrade, I.A.P

SM: Shay Moran is a Robert J. Shillman Fellow; he acknowledges support by ISF grant 1225/20, by BSF grant 2018385, by an Azrieli Faculty Fellowship, by Israel PBC-VATAT, by the Technion Center for Machine Learning and Intelligent Systems (MLIS), and by the the European Union (ERC, GENERALIZATION, 101039692). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research CouncilExecutive Agency. Neither the European Union nor the granting authority can be held responsible for them.

1. Introduction

In infinite model theory, models give rise to types, which describe elements finitely approximated in the model but possibly not realized there. In learning theory, some learning algorithms (called improper learners) necessarily choose hypotheses outside the given class, which in various examples arise naturally from notions of dimension or of majority. Motivated jointly by these ideas we develop a quantitative notion of “closure under aggregation” which we call $\epsilon$-saturation.

These definitions make sense for any graph or hypothesis class a priori. However, a characteristic of Littlestone, or stable, classes is that every finite subset of elements or hypotheses contains a “strongly opinionated aggregation” ($\epsilon$-good or $\epsilon$-excellent set) of linear size. So this is our central case. Let $\ell:=\operatorname{Ldim}(\mathcal{H})\geq d:=\operatorname{VC}(\mathcal{H})$ (defined in 3.13, 3.15 below). We define and characterize the $\epsilon$-saturation $\mathcal{H}_{\infty}(\epsilon)$ of a Littlestone class $\mathcal{H}$ for any $\epsilon<1/2$, and we prove that $\epsilon$ modulates the complexity of $\mathcal{H}_{\infty}(\epsilon)$ in a strong way. When $\epsilon<1/(2^{\ell+2})$, $\operatorname{Ldim}(\mathcal{H}_{\infty}(\epsilon))=\operatorname{Ldim}(\mathcal{H})=\ell$. For some numerical constant $C<20$, if $\epsilon<1/(Cd)$, $\mathcal{H}_{\infty}(\epsilon)$ has possibly larger, but finite, Littlestone dimension. When $1/(d+1)<\epsilon<1/2$, $\mathcal{H}_{\infty}(\epsilon)$ may have unbounded VC dimension. We then develop a parallel theory for stable graphs with the more powerful notion of $\epsilon$-excellent replacing $\epsilon$-good.

The proofs involve theorems and methods from combinatorics, statistical learning, logic, Stable Regularity, and Littlestone Minimax.

Contents

1. 1 Introduction
2. 2 Notation
3. 3 Preliminaries: good, excellent, stable, Littlestone
4. 4 Definition of $\epsilon$-saturation for $\mathcal{H}$
5. 5 Preservation of dimension for small $\epsilon$
6. 6 Characterization of $\epsilon$-saturation for Littlestone classes
7. 7 Intermediate $\epsilon$
8. 8 Conclusions for Littlestone classes
9. 9 Definition of $\epsilon$-saturation for $\mathcal{G}$
10. 10 Characterization of $\epsilon$-saturation for stable graphs
11. 11 Examples
12. 12 Examples for large $\epsilon$
13. 13 Open questions

2. Notation

We will deal with several closely related kinds of objects: graphs, bipartite graphs, and hypothesis classes. These need not necessarily be finite, though some constructions of the paper ask that they are at most countable. To fix notation,

• •

a graph $G=(V,E)$ has a vertex set and an edge set, $|G|$ means the size of the vertex set, and $E$ is symmetric and irreflexive (i.e., edges are not directed and there are no loops from a vertex to itself).

• •

we may write “$a\in G$” for $a\in V(G)$ or “$A\subseteq G$” for $A\subseteq V(G)$.

• •

for edges and nonedges, we may write “$a\sim b$” to abbreviate $(a,b)\in E$ and “$a\not\sim b$” to abbreviate $(a,b)\notin E$.

• •

a bipartite graph $G=(V_{1}\cup V_{2},E)$ is given with a partition of the vertex set, and here $E\subseteq V_{1}\times V_{2}$ is neither symmetric nor irreflexive.

• •

a hypothesis class can be denoted either $\mathcal{H}$ or $(X,\mathcal{H})$, where $X$ is a set and $\mathcal{H}$ is a set of subsets of $X$. It is also usual to identify sets $h\in\mathcal{H}$ with their characteristic functions, writing $h(a)=1$ for $a\in h$ and $h(a)=0$ for $a\notin h$.

• •

a hypothesis class $\mathcal{H}$ can be seen as a bipartite graph in the natural way, letting $X$ and $\mathcal{H}$ be the two sides of the partition of the vertex set and putting an edge between $a\in X$ and $h\in\mathcal{H}$ if and only if $h(a)=1$.

Notation used locally in sections of the paper will be introduced as needed.

3. Preliminaries: good, excellent, stable, Littlestone

In this section we briefly review and motivate good and excellent sets and stable graphs following [20], which gives one motivation for our work, and the parallel ideas in Littlestone classes. For further details, see [20], [19], [28], [17]. Note: in the full generality of the paper below, we will allow weighted good and excellent sets; but for this introductory section, the main ideas are communicated without this extra notation. The reader familiar with the area may refer back to this section as needed, or may look ahead to §11 for examples relevant to the paper.

Good sets and excellent sets were introduced by Malliaris and Shelah in [20] in order to prove the stable regularity lemma, which is a strong version of Szemerédi regularity which holds for stable graphs (see below). In this introduction, good and excellent sets are always finite, although the graph $G$ need not be. Informally, good sets are almost-atoms:

Definition 3.1** (Good).**

Given a graph $G$, $\epsilon>0$, and a finite $B\subseteq G$,

[TABLE]

if for any $a\in G$, either $\{b\in B:a\sim b\}$ or $\{b\in B:a\not\sim b\}$ has size $<\epsilon|B|$. In the first case,111Here “$\mathbf{t}$” is for “truth value,” expressing the majority opinion. write $\mathbf{t}(a,B)=1$, and in the second, $\mathbf{t}(a,B)=0$.

For any $\epsilon$-good set $B\subseteq G$ and any $a\in G$, $B$ behaves like a ‘virtual’ element in the sense that a ‘virtual’ edge or nonedge to $a$ is well defined222The astute reader may wonder about symmetry in the inputs of $\mathbf{t}$, which we address presently. (according to $\mathbf{t}(a,B)$). The following definition, also from [20], allows for a similar conclusion between virtual elements.333If $A$ and $B$ are $\epsilon$-good, they are already in some sense close to being a regular pair, at the cost of changing $\epsilon$; as we prefer to keep $\epsilon$ fixed, excellence allows for a stronger conclusion.

Definition 3.2** (Excellent).**

Given a graph $G$, $\epsilon>0$, $\delta>0$, and a finite $A\subseteq G$,

[TABLE]

if for every $\delta$-good $B\subseteq G$, either $\{a\in A:\mathbf{t}(a,B)=0\}$ or $\{a\in A:\mathbf{t}(a,B)=1\}$ has size $<\epsilon|A|$. In the first case, write $\mathbf{t}(A,B)=1$, and in the second, $\mathbf{t}(A,B)=0$. When $\epsilon=\delta$, write simply

[TABLE]

Excellent sets are good, but the reverse need not be true: see §12.2 below. The existence of good and excellent sets is characteristic of stable graphs, so-called because of stability in model theory, an important concept in Shelah’s classification theory [26]. To explain this, first recall the connection of various combinatorial definitions:

Definition 3.3** (Stable).**

A graph $G$ contains a $k$-half graph if there exist disjoint sets of vertices

[TABLE]

$G$* is called $k$-stable, or sometimes $k$-edge stable for emphasis, if it contains no $k$-half graphs.444Not all half-graph edges are specified, so $k$-stable forbids a family of configurations.*

Convention 3.4**.**

We say “$G$ is a stable graph” to mean that $G$ is $k$-stable for some finite $k$.

The next definition is also a finite version of a configuration from Shelah’s classification theory, used to sound out the complexity of a formula [such as the edge relation $E(x,y)$] by asking whether the family of instances of the formula [such as the set of neighborhoods] allowed for a certain kind of repeated partitioning. We say “special tree” after [20], others say Shelah tree or Hodges tree.555The motivation for these names will become evident shortly. Recall that in set theoretic notation, $2=\{0,1\}$, so ${{}^{m>}2}$ is the set of binary sequences of length strictly less than $m$, ${{}^{m}2}$ is the set of binary sequences of length exactly $m$, and $\eta^{\smallfrown}\langle t\rangle\trianglelefteq\nu$ means that the string $\eta$ followed by $t$ is an initial segment of $\nu$.

Definition 3.5** (Special trees).**

A graph $G$ contains a special tree of height $m$ if there exist disjoint sets of vertices $\{a_{\eta}:\eta\in{{}^{m>}2}\}$, $\{b_{\nu}:\nu\in{{}^{m}2}\}\subseteq G$, so that:

• •

if $\eta^{\smallfrown}\langle 0\rangle\trianglelefteq\nu$, then $a_{\eta}\not\sim b_{\nu}$

• •

if $\eta^{\smallfrown}\langle 1\rangle\trianglelefteq\nu$, then $a_{\eta}\sim b_{\nu}$

$($and there are no constraints on other edges$)$.

In Shelah’s classification theory, one of the consequences of his celebrated Unstable Formula Theorem ([26], Theorem II.2.2, circa 1978) is that special trees and half-graphs are either both bounded or both unbounded.666That is, in the class of models of a complete first-order theory, in the bipartite graph given by fixing a first-order formula $\varphi(\bar{x};\bar{y})$ and connecting two tuples of the correct length precisely when $\varphi$ holds, special trees and half-graphs are either both bounded or both unbounded. For a discussion of Shelah’s unstable formula theorem and learning theory, see [18]. Around 1981, Hodges looked for a quantitative version, and proved (in our language) that there is a log bound in each direction. In the next Fact, (1) is essentially sharp but (2) is not known to be. Given its role in our argument below, we sketch a proof.

Fact 3.6** (Hodges-Shelah lemma, cf. [12]).**

In a graph or bipartite graph $G$:

1. (1)

if there is a $2^{m+1}$-half graph, there is an $m$-tree 2. (2)

if there is a $2^{k+2}-2$-special tree, there is a $k$-half graph.

Proof Sketch.

First, from a $k$-half graph $a_{1},\dots,a_{k},b_{1},\dots,b_{k}$, we find a tree of height about $\log k$. We will let the $b$’s be the leaves of our tree and choose $a$’s for the internal nodes. For the root $a_{\emptyset}$, we choose the midpoint of the $a$’s, call it $a_{\lceil k/2\rceil}$, noting that this element connects to the right half of the $b$’s and not to the left half, as befits the root of the tree. Continue inductively, choosing the $2^{\ell}$ internal nodes needed for level $\ell$ as the midpoints of the segments with endpoints $\lceil ki/2^{\ell}\rceil$.

From an $m$-tree $A=\{a_{\eta}:\eta\in{{}^{m>}2}\}$, $B=\{b_{\nu}:\nu\in{{}^{m}2}\}$ we can obtain a half-graph of length about $\log m$ by induction. There are two preliminary steps. First, by induction on $\ell$, we define what it means to be a subtree of height $\ell$. A subtree of height [math] is a leaf, and a subtree of height $\ell+1$ consists of an internal node $a_{\eta}$ which has two distinct extensions $a_{\rho}$ and $a_{\nu}$, which are each the root of a subtree of height $\ell$ (here $\eta$ is the common initial segment of $\rho$ and $\nu$, but $\rho$ and $\nu$ need not have the same length). Second, one proves a Ramsey lemma saying that if we 2-color the internal nodes of an $m$-tree then there is a monochromatic subtree of height about $m/2$.

Now let us choose a first pair of elements for our half-graph. (We will either be choosing $a_{1},b_{1}$ or $a_{k},b_{k}$.) Start by choosing any $b\in B$, which induces a 2-coloring of $A$ according to whether or not $b\sim a$. By the Ramsey lemma we can find a large monochromatic subtree. Call it $A_{*}$, and suppose that $b\sim a$ for all $a\in A_{*}$. In this case our $b$ will become $b_{k}$. Define $a_{k}$ to be the root of the tree $A_{*}$. Let $A_{**}$ be the subtree whose root is the left successor of $a_{k}$ in $A_{*}$, and let $B_{**}$ be its set of leaves. Now we have found $a_{k},b_{k}$, and we have a slightly smaller tree $A_{**},B_{**}$ with the property that $b_{k}\sim a$ for all $a\in A_{**}$, and $a_{k}\not\sim b$ for all $b\in B_{**}$. Note also that had our Ramsey lemma returned $\not\sim$, we would have set $b_{1}=b$, $a_{1}$ to be the root of $A_{*}$, $A_{**}$ to be the subtree whose root is the right successor of $a_{1}$ in $A_{*}$, and $B_{**}$ its leaves . We would then have found $a_{1},b_{1}$ and a slightly smaller tree $A_{**},B_{**}$ with the property that $b_{1}\not\sim a$ for all $a\in A_{**}$, and $a_{1}\sim b$ for all $b\in B_{**}$. Either way, we can clearly focus on $A_{**},B_{**}$ and repeat. ∎

Conclusion 3.7**.**

If $G$ is a $k$-stable graph, hence has no $k$-half graphs, there is $m=m(k)\leq 2^{k+2}-2$ such that $G$ has no $m$-special trees.

This connection of orders and trees was leveraged by Malliaris and Shelah in the stable regularity lemma to prove existence of good and excellent sets. These proofs will be central to our work in this paper so we review them here. Since sets of size one are good, of course 3.9 implies 3.8 and the inclusion of both proofs is pedagogical; also, if the $X$ with which we begin is too small, say $|X|<2^{m}$, we can choose any singleton and finish.

Claim 3.8** (Good sets in stable graphs, [20]).**

Let $G$ be a $k$-stable graph, and let $m=m(k)$ be from 3.7. Then for any $0<\epsilon<\frac{1}{2}$, and any finite $X\subseteq G$, there is $Y\subseteq X$, $|Y|\geq\epsilon^{m}|X|$ such that $Y$ is $\epsilon$-good.

Proof.

We proceed by induction777If $X$ is not $\epsilon$-good, letting $a_{\emptyset}\in G$ be such that $B_{\langle 0\rangle}:=\{b\in X:b\not\sim a_{\emptyset}\}$ and $B_{\langle 1\rangle}:=\{b\in X:b\sim a_{\emptyset}\}$ are both nonempty of size $\geq\epsilon|X|$ starts building a tree. on $\ell<m$. Let $X_{\emptyset}=X$. For each $\eta\in{{}^{\ell}2}$, if $X_{\eta}$ is not $\epsilon$-good, let $a_{\eta}\in G$ be such that $X_{\eta^{\smallfrown}\langle 0\rangle}:=\{x\in X_{\eta}:x\not\sim a_{\eta}\}$ and $X_{\eta^{\smallfrown}\langle 1\rangle}:=\{x\in X_{\eta}:x\sim a_{\eta}\}$ are both nonempty of size $\geq\epsilon|X_{\eta}|$. Suppose the construction continues through stage $m-1$ so we have defined $\{X_{\rho}:\rho\in{{}^{m}2}\}$ (which are by construction disjoint) and $\{a_{\nu}:\nu\in{{}^{m>}2}\}$. By hypothesis each $X_{\rho}\neq\emptyset$, so we may choose $x_{\rho}\in X_{\rho}$ for each $\rho\in{{}^{m}2}$. Then $\{a_{\nu}:\nu\in{{}^{m>}2}\}$, $\{x_{\rho}:\rho\in{{}^{m}2}\}$ form an $m$-tree, contradiction. So one of the $X_{\eta}$’s (for $\eta$ of length $\leq m-1$) must have been $\epsilon$-good, and its size will be $\geq\epsilon^{m}|X|$. ∎

Lemma 3.9** (Excellent sets in stable graphs, [20]).**

Let $G$ be a $k$-stable graph, and let $m=m(k)$ be from 3.7. Then for any $0<\epsilon<\frac{1}{2^{m}}$, and any finite $X\subseteq G$, there is $Y\subseteq X$, $|Y|\geq\epsilon^{m}|X|$ such that $Y$ is $\epsilon$-excellent.

Proof.

We also proceed by induction on $\ell<m$. Let $X_{\emptyset}=X$. For each $\eta\in{{}^{\ell}2}$, if $X_{\eta}$ is not $\epsilon$-excellent, let $A_{\eta}\subseteq G$ be an $\epsilon$-good set such that $X_{\eta^{\smallfrown}\langle 0\rangle}:=\{x\in X_{\eta}:\mathbf{t}(x,A_{\eta})=0\}$ and $X_{\eta^{\smallfrown}\langle 1\rangle}:=\{x\in X_{\eta}:\mathbf{t}(x,A_{\eta})=1\}$ are both nonempty of size $\geq\epsilon|X_{\eta}|$. Suppose the construction continues through stage $m-1$ so we have defined $\{X_{\rho}:\rho\in{{}^{m}2}\}$ (which are by construction disjoint) and $\{A_{\nu}:\nu\in{{}^{m>}2}\}$.

We now have a ‘virtual’ tree from which we would like to extract an ‘actual’ special tree. Since $X_{\rho}\neq\emptyset$ for every $\rho\in^{m}2$, we first choose any $x_{\rho}\in X_{\rho}$ for each such $\rho$. For each $\nu\in{{}^{m>}2}$, let $\mathbf{t}_{\rho}$ be such that $\mathbf{t}(x_{\rho},A_{\nu})=\mathbf{t}_{\rho}$. This can be considered as a virtual edge, and we want to find an actual element $a_{\nu}\in A_{\nu}$ which has an actual edge of the correct kind to $x_{\rho}$, for all $\rho$ extending $\nu$. More precisely, the set $\{\rho\in{{}^{m}2}:\nu\triangleleft\rho\}$ has size $\leq 2^{m}$. For any such $\rho$, the set $\{a\in A_{\nu}:\mathbf{t}(x_{\rho},a)\neq\mathbf{t}(x_{\rho},A_{\nu})\}$ of incorrect elements has size has size $<\epsilon|A_{\nu}|$. By a union bound and the upper bound on the value of $\epsilon$, after each $x_{\rho}$ has ruled out its small set, some $a_{\nu}$ remains. Then $\{a_{\nu}:\nu\in{{}^{m>}2}\}$, $\{x_{\rho}:\rho\in{{}^{m}2}\}$ form a depth $m$ special tree, contradiction. So one of the $X_{\eta}$’s (for $\eta$ of length $\leq m-1$) must have been $\epsilon$-excellent, and its size is at least $\epsilon^{m}|X|$. ∎

Note that this result can be extended to any $\epsilon<\frac{1}{2}$ by a theorem of Malliaris and Moran [17]. However, in that result the guarantee on the size of the excellent set, while still linear, may be smaller. The proof is beyond the scope of this introduction and involves multiplicative-weights algorithms.

Conclusion 3.10** ([17] Claim 2.11, in our language).**

For a graph $G$, the following are equivalent:

1. (1)

$G$* is stable,* 2. (2)

for every $\epsilon<1/2$ there is a constant $c=c(\epsilon)>0$ such that for every finite $A\subseteq G$ there is $B\subseteq A$, $|B|\geq c|A|$, such that $|B|$ is $\epsilon$-good.

As mentioned above, a hypothesis class $(X,\mathcal{H})$ can naturally be seen as a bipartite graph. The natural analogue in learning of half-graphs is called threshold dimension (as was used in Alon-Livni-Malliaris-Moran [4]). Note that this dimension is a priori one more than stability as $\operatorname{Thr}(G)=k$ means the largest half-graph has size $k$, i.e., $G$ is $(k+1)$-stable.

Definition 3.11** (Thresholds).**

The threshold dimension of a hypothesis class $\mathcal{H}$, $\operatorname{Thr}(\mathcal{H})$, is defined to be the greatest natural number $\ell$, if it exists, such that there exist $x_{1},\dots,x_{\ell}\in X$ and $h_{1},\dots,h_{\ell}\in\mathcal{H}$ with $h_{i}(x_{j})=1$ iff $i<j$. If no such $\ell$ exists888One can also define ordinal threshold dimension, see REFERENCE. Note that in this definition, $\operatorname{Thr}(\mathcal{H})=\infty$ does not distinguish between the case where there is a single infinite half-graph, and arbitrarily large finite half-graphs, $\operatorname{Thr}(\mathcal{H})=\infty$.

The natural analogue in learning of special trees are Littlestone mistake trees, as pointed out by Chase and Freitag [7], who used this to give new examples of Littlestone classes. Note that here too we record the largest $d$ where a tree occurs rather than the smallest $d$ where it does not; also, following convention, the root of a mistake tree is at level $1$, not level [math].

Definition 3.12** (Littlestone mistake tree).**

Let $(X,\mathcal{H})$ be a hypothesis class. A Littlestone mistake tree of height $d$ is a full binary tree whose internal nodes are named by elements of X, and which is shattered by $\mathcal{H}$ in the following sense.

1. (a)

*We can represent the process of traversing a root-to-leaf path (i.e., a branch) in the tree as a sequence of pairs $\langle(x_{i},y_{i}):1\leq i\leq d\rangle$ where $(x_{i},y_{i})$ records that the name of the current node is $x_{i}\in X$ and from this node we travel right *$($*if y_{i}=1$$) or left *$($if y_{i}=0$$). 2. (b)

We say $h\in\mathcal{H}$ realizes a given branch $\langle(x_{i},y_{i}):1\leq i\leq d\rangle$ if for all $1\leq i\leq d$, $h(x_{i})=y_{i}$. 3. (c)

We say that $\mathcal{H}$ shatters the tree if every branch is realized by some $h\in\mathcal{H}$.

Definition 3.13** (Littlestone).**

The Littlestone dimension of a hypothesis class, $\operatorname{Ldim}(\mathcal{H})$, is defined to be the greatest natural number $d$, if it exists, so that there exists a Littlestone mistake tree of height $d$ which is shattered by $\mathcal{H}$. If no such $d$ exists, we set $\operatorname{Ldim}(\mathcal{H})=\infty$. $\mathcal{H}$ is a Littlestone class if $\operatorname{Ldim}(\mathcal{H})<\infty$.

Littlestone classes are well-studied in the learning literature, and are characterized by being online learnable (see Littlestone [16] and Ben-David, Pál, Shalev-Schwartz [5]). For more on Littlestone classes and online learning see [25] §§18.1-18.2.999Note also that $\operatorname{Ldim}(\mathcal{H})=\infty$ a priori does not distinguish between shattered trees of arbitrarily large finite height and shattered trees of infinite height. See CITE

In what follows, we shall use the analogue of goodness, excellence, and the Hodges-Shelah lemma in hypothesis classes in the natural way. This takes up a line of work beginning with Alon-Livni-Malliaris-Moran and Bun-Livni-Moran showing that the property of being Littlestone, i.e. stable, gave a qualitative characterization of when a hypothesis class was differentially-privately PAC learnable. This connection of stability/Littlestone/online learning to statistical learning was part of the motivation for the present paper.

For completeness, recall the Vapnik-Chervonenkis dimension of a class $(X,\mathcal{H})$.

Definition 3.14** (Shattering).**

In a hypothesis class $(X,\mathcal{H})$, say a set $S\subseteq X$ is shattered if, for every $S^{\prime}\subseteq S$, there exists $h\in\mathcal{H}$ such that $\{x\in X:h(x)=1\}\cap S=S^{\prime}$.

Definition 3.15** (VC dimension).**

The Vapnik-Chervonenkis dimension of a hypothesis class, $\operatorname{VC}(\mathcal{H})$, is defined to be the greatest natural number $m$, if it exists, so that there exists a set $S$ of size $m$ which is shattered by $\mathcal{H}$. If no such $m$ exists, we set $\operatorname{VC}(\mathcal{H})=\infty$. $\mathcal{H}$ is called a VC class if $\operatorname{VC}(\mathcal{H})<\infty$.

Note that $\operatorname{Ldim}\geq\operatorname{VC}$ for every hypothesis class $\mathcal{H}$ (and in the graph case, for every graph $\mathcal{G}$). This is because shattering a set in the way described above corresponds to a mistake tree with the same structure as a Littlestone mistake tree, with the additional condition that all internal nodes which lie in the same level must be the same element of $X$.

Finally, it will be useful to define the dual of a hypothesis class. Informally, any hypothesis class corresponds to a bipartite graph with $X$ on the left, $\mathcal{H}$ on the right, and an edge between $x$ and $h$ if and only if $h(x)=1$. The dual class corresponds to the bipartite graph we would get by swapping the left and right sides without changing any edges. More formally:

Definition 3.16** (Dual hypothesis class).**

For a hypothesis class $(X,\mathcal{H})$, define the dual hypothesis class $\operatorname{dual}(\mathcal{H})$ to be $(Y,\mathcal{G})$ where $Y=\mathcal{H}$ is the domain and $\mathcal{G}=\{g_{x}:x\in X\}$ is a set of functions from $\mathcal{H}$ to $\{0,1\}$ given by: $g_{x}(h)=1$ if and only if $h(x)=1$.

$\mathcal{H}$ is a VC class if and only if $\operatorname{dual}(\mathcal{H})$ is a VC class, possibly of larger dimension, and $\mathcal{H}$ is Littlestone if and only if $\operatorname{dual}(\mathcal{H})$ is Littlestone, possibly of larger dimension. For the first fact, see e.g. Lemma 10.3.4 in [22]. The second fact is proved by using a Hodges-Shelah tree to say that Littlestone dimension is finite iff Threshold dimension is finite:

Conclusion 3.17** (Hodges-Shelah lemma translated to hypothesis classes).**

Given a hypothesis class $(X,\mathcal{H})$:

1. (1)

If $\operatorname{Thr}(\mathcal{H})\geq 2^{m+1}$, then $\operatorname{Ldim}(\mathcal{H})\geq m$. 2. (2)

If $\operatorname{Ldim}(\mathcal{H})\geq 2^{k+2}-2$, then $\operatorname{Thr}(\mathcal{H})\geq k$.

since Threshold dimension is clearly symmetric between a class and its dual.

4. Definition of $\epsilon$-saturation for $\mathcal{H}$

In this section we give the central definitions of the paper for hypotheses classes, corresponding to saturation and to types. Graphs are considered in §9.

These definitions work when $\mathcal{H}$ is finite, but they don’t require it. The sets of aggregates we take, however, are always finite.

Convention 4.1**.**

$(X,\mathcal{H})$, which we may abbreviate $\mathcal{H}$, will usually denote a Littlestone class, with

• •

finite Littlestone dimension $\ell$

• •

and hence finite VC dimension $d$.

$\mathcal{H}$* can be finite, but need not be.*

We will state “$\mathcal{H}$ is Littlestone” when it is used in the main definitions and theorems, and will keep the convention of $\ell$ and $d$.

Convention 4.2**.**

Below, $\epsilon$ and $\delta$ $($possibly with subscripts or other decorations$)$ are always in the interval $(0,\frac{1}{2})_{\mathbb{R}}$, subject to further restrictions as stated.

Notation 4.3**.**

A notational convention: we think of integers set-theoretically, so $n=\{0,\dots,n-1\}$ and “$i<n$” means $i$ ranges over the $n$ values $0,\dots,n-1$.

In the present paper, we allow good and excellent sets to be weighted. By convention we ask the weights to be positive, although we could allow zero weights at the cost of modifying many instances of “thus there exists $g$” to “thus there exists $g$ of nonzero weight” below.

Definition 4.4**.**

Given $\epsilon$ and $\mathcal{H}$, an $\epsilon$-hypothesis $($or a weighted $\epsilon$-good set$)$ of $\mathcal{H}$ is a finite set of pairs:

[TABLE]

where $k\in\mathbb{N}$, and:

1. (1)

$\{h_{i}:i<k\}\subseteq\mathcal{H}$, and $\langle h_{i}:i<k\rangle$ is without repetition. 2. (2)

each $\gamma_{i}\in(0,1]_{\mathbb{R}}$, and $\sum\{\gamma_{i}:i<k\}=1$. 3. (3)

($$\epsilon-goodness$)$* For each $x\in X$, for some $\mathbf{t}=\mathbf{t}(x,H)=\mathbf{t}_{H}(x)\in\{0,1\}$,*

[TABLE]

Here $\mathbf{t}$ represents the “truth value” of the majority opinion between $x$ and $H$.

Remark 4.5**.**

As the definition records, any $\epsilon$-hypothesis $H$ determines a function $\mathbf{t}_{H}:X\rightarrow\{0,1\}$. Say that $H$ represents $\mathbf{t}_{H}$. A priori, $\mathbf{t}_{H}$ need not be in $\mathcal{H}$. Also, a priori, different $\epsilon$-hypotheses may represent the same function.

We will use several different notions of approximation. Since these are well-studied we keep the usual names:

Definition 4.6**.**

Suppose we are given $\mathcal{H}$ and $f:X\rightarrow\{0,1\}$. Say $f$ is:

1. (1)

$k$-realizable* in $\mathcal{H}$, for a fixed $k\in\mathbb{N}$, when for every $Y\subseteq X$, $|Y|\leq k$, there exists $h\in\mathcal{H}$ such that $h\upharpoonright Y=f\upharpoonright Y$.* 2. (2)

finitely realizable* in $\mathcal{H}$ if $f$ is $k$-realizable in $\mathcal{H}$ for every finite $k$.* 3. (3)

realized* if $f\in\mathcal{H}$.*

Write $\mathcal{H}^{\operatorname{k-real}}$ for the set of functions $f:X\rightarrow\{0,1\}$ which are $k$-realizable in $\mathcal{H}$, and $\mathcal{H}^{{\operatorname{(<\omega)-real}}}=\bigcap\{\mathcal{H}^{\operatorname{k-real}}:k<\omega\}$ for the set of finitely realizable functions.

Remark 4.7**.**

Note that in the definition above, $\mathcal{H}^{{\operatorname{k-real}}}$ could be defined as well if $k$ is a positive real number. Then, $\mathcal{H}^{{\operatorname{k-real}}}=\mathcal{H}^{\lfloor k\rfloor\operatorname{-real}}$. This will be used in later sections.

The next will play a key role. To see why 4.8 is preferable to finite realizability here, consider that when $X$ is finite, the only finitely realizable functions are those already in $\mathcal{H}$, but for $\epsilon>1/|X|$ there may be $\epsilon$-types which are not realized, as in §11.2 below.

Definition 4.8** ($\epsilon$-types).**

Suppose we are given $\mathcal{H}$, $0<\epsilon<1/2$, and $f:X\rightarrow\{0,1\}$. We call $f$ an $\epsilon$-type of $\mathcal{H}$, or just $\epsilon$-type, if $f$ is finitely realized in the class

[TABLE]

In a slogan, the function $f$ is an $\epsilon$-type if its restriction to any finite set agrees with some $\epsilon$-hypothesis (that is, with some finite weighted $\epsilon$-good set of hypotheses).

The central definition of this paper, in the case of hypothesis classes, is:

Definition 4.9** (The $\epsilon$-saturation of $\mathcal{H}$).**

Fix $0<\epsilon<1/2$ and $(X,\mathcal{H})$ a hypothesis class. Define by induction on $n\leq\omega$ a hypothesis class $\mathcal{H}_{n}(\epsilon)$ over $X$:

1. (1)

$\mathcal{H}_{0}(\epsilon)=\mathcal{H}$. 2. (2)

\mathcal{H}_{n+1}(\epsilon)=\mathcal{H}_{n}\cup\{\mathbf{t}_{H}:\mbox{ H$is an$\epsilon$-hypothesis of$\mathcal{H}_{n}}\}. 3. (3)

$\mathcal{H}_{\infty}(\epsilon):=\mathcal{H}_{\omega}(\epsilon)=\bigcup\{\mathcal{H}_{n}(\epsilon):n<\omega\}$.

We call $\mathcal{H}_{\infty}(\epsilon)$ “the $\epsilon$-saturation of $\mathcal{H}$.”

Note that in condition (2) we define $\mathcal{H}_{n+1}$ as a set of functions, not names for functions; so in particular, if $H_{1}$ and $H_{2}$ are two different $\epsilon$-hypotheses (that is, different finite weighted good sets) but $\mathbf{t}_{H_{1}}=\mathbf{t}_{H_{2}}$ as functions, we only add one copy of this function. (In particular, we do not re-add copies of functions already in $\mathcal{H}$.) In later calculations, it will not matter which representation we choose.

The words “saturation” and “type” come from model theory. In model theoretic language, saturation describes a structure where all types of a suitable size are realized. So this notation suggests a connection of $\epsilon$-saturation and $\epsilon$-types, which will be justified in due course.

Remark 4.10**.**

In Definition 4.9, note $X$ stays fixed. If we were to increase both $X$ and $H$ by adding majorities, then in order to retain total functions, we would need (in the language of stable regularity) to upgrade “$\epsilon$-good” in the definition of $\epsilon$-hypothesis above to “$\epsilon$-excellent.” This will be a consideration when we deal with graphs rather than hypothesis classes, see 9.4 below.

Remark 4.11**.**

In Definition 4.9 we could have defined $\mathcal{H}_{\alpha}(\epsilon)$ for ordinals $\alpha$ by transfinite induction, but this will not be used here.

Remark 4.12**.**

Observe that as $n\rightarrow\infty$, the set of $\epsilon$-types may change even though $X$ stays the same, because the $\epsilon$-hypotheses available in $\mathcal{H}_{n}(\epsilon)$ to witness finite realizability may be changing. See for instance §11.5.

5. Preservation of dimension for small $\epsilon$

This section proves Theorem 5.6 which says that Littlestone, threshold, and VC dimension are preserved from $\mathcal{H}$ to $\mathcal{H}_{\infty}(\epsilon)$ provided $\epsilon$ is sufficiently small relative to the dimension in question, by applying ideas from stable regularity (as outlined in §3) to weighted good and excellent sets in the natural way. The theorem will rely on several lemmas.

For orientation, recall “$\operatorname{Ldim}\geq\ell$” and “$\operatorname{VC}\geq d$” and “$\operatorname{Thr}\geq k$” are witnessed by the existence of certain finite configurations: the nodes and leaves of a shattered tree of height $\ell$ in the first case, the shattered set of size $d$ (that is, $d$ elements and $2^{d}$ hypotheses realizing all possible functions over them) in the second, and a half-graph of length $k$ in the third. (We will also use that $VC$ dimension can be witnessed by a shattered tree where all the labels at a given level are the same.) So in each case, the Littlestone, VC, or threshold dimension of $\mathcal{H}_{n}$ is nondecreasing with $n$. Also, since these are finite configurations, if the given configuration exists in $\mathcal{H}_{\infty}$ then already it exists in $\mathcal{H}_{n+1}$ for some $n$.

We will say “special tree” rather than Littlestone tree to emphasize the parallelism, since in some cases we will want to choose the nodes from the $X$ side and the leaves from the hypothesis side, and in other cases the reverse.

Remark 5.1**.**

Recall our set-theoretic notation: for $s\in\mathbb{N}$, “$i<s$” means that $i$ takes the $s$ values $0,\dots,s-1$. Recall also from 3.5 that ${{}^{s>}2}$ is the set of binary sequences of length strictly less than $s$, and ${{}^{s}2}$ is the set of binary sequences of length $s$.

For Lemmas 5.2-5.5 let $(X,\mathcal{H})$ be an arbitrary but fixed hypothesis class. Here we can allow “$\epsilon\leq\dots$” instead of “$\epsilon<\dots$” just because the definition of $\epsilon$-hypothesis has a strict inequality on the smaller side.

Lemma 5.2**.**

For any $s\in\mathbb{N}$ and $\epsilon\leq 1/s$, if there is a special tree of height $s$ with the internal nodes chosen from $X$ and the leaves chosen from $\mathcal{H}_{\infty}$, then there is a special tree of height $s$ with the same internal nodes, and the leaves chosen from $\mathcal{H}$.

Proof.

Consider a special tree given by

[TABLE]

Let $n<\omega$ be minimal such that $\{y_{\rho}:\rho\in{{}^{s}2}\}\subseteq\mathcal{H}_{n+1}$. For each $y_{\rho}$ choose an $\epsilon$-hypothesis $H_{\rho}$ of $\mathcal{H}_{n}$ representing the same function over $X$ as $y_{\rho}$. So we now have $\langle x_{\eta}:\eta\in{{}^{s>}2}\rangle$ and $\langle H_{\rho}:\rho\in{{}^{s}2}\rangle$ with $\mathbf{t}(H_{\rho},x_{\eta})=1$ if $\eta^{\smallfrown}\langle 1\rangle\trianglelefteq\rho$ and $\mathbf{t}(H_{\rho},x_{\eta})=0$ if $\eta^{\smallfrown}\langle 0\rangle\trianglelefteq\rho$. For each $\rho\in{{}^{s}2}$, define the error set

[TABLE]

Observe that $\{\eta:\eta\triangleleft\rho\}$ has size $s$ and that for each $\eta$, the “error weight” is $<\epsilon$. In other words, $\sum\{\gamma_{i}:(h_{i},\gamma_{i})\in E_{\rho}\}<\epsilon s\leq(1/s)s=1$. So there is some $(h,\gamma)\in H_{\rho}\setminus E_{\rho}$. Set $a_{\rho}:=h$. Having thus defined each $a_{\rho}$, we have found a special tree with internal nodes $\langle x_{\eta}:\eta\in{{}^{s>}2}\rangle$ from $X$ and leaves $\langle a_{\rho}:\rho\in{{}^{s}2}\rangle$ from $\mathcal{H}_{n}$. Either $n=0$, or we contradict our choice of $n$, so either way we are done. ∎

Lemma 5.3**.**

For any $s\in\mathbb{N}$ and $\epsilon\leq 1/2^{s}$, if there is a special tree of height $s$ where the internal nodes are from $\mathcal{H}_{\infty}$ and the leaves are from $X$, there is also a special tree of height $s$ with internal nodes from $\mathcal{H}$ and the same leaves from $X$.

Proof.

Consider $\langle z_{\eta}:\eta\in{{}^{s>}2}\rangle$ from $\mathcal{H}_{\infty}$ and $\langle x_{\rho}:\rho\in{{}^{s}2}\rangle$ from $X$ which form a special tree. Let $n<\omega$ be minimal so that $\{z_{\eta}:\eta\in{{}^{s>}2}\}\subseteq\mathcal{H}_{n+1}$. For each $\eta\in{{}^{s>}2}$ choose an $\epsilon$-hypothesis $H_{\eta}$ of $\mathcal{H}_{n}$ representing the same type over $X$ as $z_{\eta}$. For orientation, this means $\mathbf{t}(H_{\eta},x_{\rho})=1$ if $\eta^{\smallfrown}\langle 1\rangle\trianglelefteq\rho$ and $\mathbf{t}(H_{\eta},x_{\rho})=0$ if $\eta^{\smallfrown}\langle 0\rangle\trianglelefteq\rho$. For each of the internal indices $\eta\in{{}^{s>}2}$ there are $\leq 2^{s}$ leaf indices $\rho\in 2^{s}$ with $\eta\trianglelefteq\rho$. For each $\eta\in{{}^{s>}2}$, define the error set

[TABLE]

Then $\sum\{\gamma_{i}:(h_{i},\gamma_{i})\in E_{\eta}\}<\epsilon 2^{s}\leq 1$ so there is some $(h,\gamma)\in H_{\eta}\setminus E_{\eta}$. Set $c_{\eta}=h$. Having chosen all such $c_{\eta}$s, $\langle c_{\eta}:\eta\in{{}^{s>}2}\rangle$ from $H_{n}$ and $\langle x_{\rho}:\rho\in{{}^{s}2}\rangle$ from $X$ give a special tree of height $s$. ∎

Lemma 5.4**.**

In the notation of the previous proof: suppose that our original tree $\langle z_{\eta}:\eta\in{{}^{s>}2}\rangle$, $\langle x_{\rho}:\rho\in{{}^{s}2}\rangle$ had the property that for all $\eta_{1},\eta_{2}\in{{}^{s>}2}$, if $\operatorname{lg}(\eta_{1})=\operatorname{lg}(\eta_{2})$ then $z_{\eta_{1}}=z_{\eta_{2}}$. Then we can ensure that for all $\eta_{1},\eta_{2}\in{{}^{s>}2}$, if $\operatorname{lg}(\eta_{1})=\operatorname{lg}(\eta_{2})$ then $c_{\eta_{1}}=c_{\eta_{2}}$.

Proof.

The property says that for each $t<s$, for every $\eta$ of length $t$ we chose the same value. For each such $t$, then, make sure to choose the same $H_{\eta}$ representing each $z_{\eta}$ of length $t$, and denote it by $H_{t}$. Then we simply let the error set for $H_{t}$ collect the wrong votes from all of the $x_{\rho}$’s, of which there are $2^{s}$-many. ∎

Lemma 5.5**.**

Suppose there is a half-graph consisting of $\langle x_{i}:i<s\rangle$ from $X$ and $\langle y_{i}:i<s\rangle$ from $\mathcal{H}_{\infty}$. If $\epsilon\leq 1/s$, there is a half-graph consisting of the same $\langle x_{i}:i<s\rangle$ from $X$ and $\langle z_{i}:i<s\rangle$ from $\mathcal{H}$.

Proof.

Again, the error set picks up an error of weight strictly less than $\epsilon$ from each of the at most $s$ elements on the opposite side. ∎

Summarizing:

Theorem 5.6** (Preservation theorems for small $\epsilon$).**

Let $\mathcal{H}$ be any hypothesis class and $s\in\mathbb{N}$.

1. (1)

If $\epsilon\leq 1/s$,

[TABLE]

and also

[TABLE] 2. (2)

If $\epsilon\leq 1/2^{s}$,

[TABLE]

and also

[TABLE] 3. (3)

Hence if $\mathcal{H}$ is a Littlestone class and $\epsilon\leq 1/(\operatorname{Ldim}(\mathcal{H})+1)$ then

[TABLE] 4. (4)

Hence if $\mathcal{H}$ is a VC class and $\epsilon\leq 1/(\operatorname{VC}(\mathcal{H})+1)$ then

[TABLE] 5. (5)

Note: regardless of $\epsilon$, if $\mathcal{H}$ is not Littlestone, $\operatorname{Ldim}(\mathcal{H})=\operatorname{Ldim}(\mathcal{H}_{\infty}(\epsilon))=\infty$, and if $\mathcal{H}$ is not a VC class, $\operatorname{VC}(\mathcal{H})=\operatorname{VC}(\mathcal{H}_{\infty}(\epsilon))=\infty$. 6. (6)

In particular, if $\operatorname{Ldim}(\mathcal{H})=\ell\in\mathbb{N}$ and $\epsilon<1/2^{\ell+2}$ then $\operatorname{Ldim}(\mathcal{H})=\operatorname{Ldim}(\mathcal{H}_{\infty}(\epsilon))$, $\operatorname{Thr}(\mathcal{H})=\operatorname{Thr}(\mathcal{H}_{\infty}(\epsilon))$, $\operatorname{Ldim}(\operatorname{dual}(\mathcal{H}))=\operatorname{Ldim}(\operatorname{dual}(\mathcal{H}_{\infty}(\epsilon)))$, $\operatorname{VC}(\mathcal{H})=\operatorname{VC}(\mathcal{H}_{\infty}(\epsilon))$ and $\operatorname{VC}(\operatorname{dual}(\mathcal{H}))=\operatorname{Ldim}(\operatorname{VC}(\mathcal{H}_{\infty}(\epsilon)))$.

Proof.

1. (1)

The first equation is Lemma 5.2. The second is also Lemma 5.2, with the extra observation that a hypothesis class $(X,\mathcal{G})$ shatters a subset $\{x_{0},\dots,x_{s-1}\}$ from $X$ if and only if we can find a special tree of height $s$ with internal nodes from $X$ and leaves from $\mathcal{G}$ where, for each $t<s$, every node at level $t$ is equal to $x_{t}$. 2. (2)

As in (1) using Lemmas 5.3 and 5.4. 3. (3)

By monotonicity, $\operatorname{Ldim}(\mathcal{H})\leq\operatorname{Ldim}(\mathcal{H}_{\infty}(\epsilon))$. Let $\ell=\operatorname{Ldim}(\mathcal{H})$. By the choice of $\epsilon$ and Lemma 5.2, if $\operatorname{Ldim}(\mathcal{H}_{\infty}(\epsilon))\geq\ell+1$ then also $\operatorname{Ldim}(\mathcal{H})\geq\ell+1$, contradiction. 4. (4)

As in (3) with $\operatorname{VC}$ in place of $\operatorname{Ldim}$. 5. (5)

By monotonicity. 6. (6)

We know that $\operatorname{Ldim}(\mathcal{H})\geq\operatorname{VC}(\mathcal{H})$ so the results on $\operatorname{Ldim}$ and $\operatorname{VC}$ of $\mathcal{H}_{\infty}(\epsilon)$ and its dual follow from the earlier statements. The minor new point is that a small $\epsilon$ in terms of $\operatorname{Ldim}(\mathcal{H})$ helps with the threshold dimension also. Let $k=\operatorname{Thr}(\mathcal{H})$. Taking the contrapositive of Conclusion 3.17(1) in the case $m=\ell+1$ we see that $k<2^{\ell+2}$ and so the contrapositive of 5.5 applies.

This completes the proof. ∎

Note: these results could have been stated in the language of $k$-realizability, which has other advantages, but then they would not as obviously extend to graphs.

Before continuing, the reader may wish to look at the section of motivating examples, §11 below.

6. Characterization of $\epsilon$-saturation for Littlestone classes

Definition 6.1**.**

Call a hypothesis class $\mathbf{H}$ $\epsilon$-saturated if every $\epsilon$-type of $\mathbf{H}$ is realized in $\mathbf{H}$.

The aim of this section is to prove:

Theorem 6.2** ($\mathcal{H}_{\infty}(\epsilon)$ is $\epsilon$-saturated).**

Suppose $\mathcal{H}$ and $\epsilon<1/2$ are given. Suppose $\mathcal{H}_{\infty}(\epsilon)$ is a Littlestone class $($hence also $\mathcal{H}$ is a Littlestone class$)$. Then every $\epsilon$-type of $\mathcal{H}_{\infty}(\epsilon)$ is realized in $\mathcal{H}_{\infty}(\epsilon)$, i.e. $\mathcal{H}_{\infty}(\epsilon)$ is $\epsilon$-saturated.

We will invoke the Littlestone Minimax Theorem, recently proved by Hanneke, Livni and Moran, which we state here in the language of the present paper.101010See [10] Corollary 7, where finite support is explicitly mentioned; also relevant for us are Lemma 11 and Lemma 12 of that paper.

Theorem A** (Littlestone Minimax Theorem; Hanneke, Livni, Moran [10]).**

*Suppose $\mathbf{H}$ is a Littlestone class and $\epsilon_{L}<1/2$. Let $g:X\rightarrow\{0,1\}$ be any function.

If $(A)$ then $(B)$.*

1. (A)

For every set $\{(x_{t},w_{t}):t<s\}$ where $\{x_{t}:t<s\}\subseteq X$, $0<w_{t}\leq 1$ for each $t<s$, and $\sum_{t}w_{t}=1$, there is $h\in\mathbf{H}$ so that

[TABLE]

Informally, “for every finitely supported probability measure on $X$ there is some $h\in\mathbf{H}$ whose error against $g$ is $<\epsilon_{L}$.” 2. (B)

For every $\epsilon_{M}$ with $\epsilon_{L}<\epsilon_{M}<1/2$, there exists a finite weighted subset $\{(h_{i},\gamma_{i}):i<m\}$ of $\mathbf{H}$ such that: for every $x\in X$,

[TABLE]

In our language, “there is an $\epsilon_{M}$-hypothesis $H$ of $\mathbf{H}$ representing $g$.”

We work towards Theorem 6.2.

Claim 6.3**.**

Assume $\mathcal{H}_{\infty}(\epsilon)$ is Littlestone. For any $n<\omega$, if $g:X\rightarrow\{0,1\}$ is finitely realized in $\mathcal{H}_{n}(\epsilon)$, then $g\in\mathcal{H}_{n+1}(\epsilon)$.

Proof.

In this case we can satisfy condition (A) of Littlestone Minimax for any $\epsilon_{L}>0$, simply by choosing $h\in\mathcal{H}_{n}(\epsilon)$ to be the function which agrees with $g$ on $\{x_{i}:i<s\}$ for some $s<\omega$. In particular, we can take $\epsilon_{M}=\epsilon$. Let $H$ be the $\epsilon$-hypothesis given by condition (B) where the $h_{i}$s belong to $\mathcal{H}_{n}(\epsilon)$. Then $\mathbf{t}_{H}$ belongs to $\mathcal{H}_{n+1}(\epsilon)$, or in other words, $g\in\mathcal{H}_{n+1}(\epsilon)$. ∎

Claim 6.4**.**

Assume $\mathcal{H}_{\infty}(\epsilon)$ is Littlestone. For any $n<\omega$, if $\mathbf{t}:X\rightarrow\{0,1\}$ is an $\epsilon$-type of $\mathcal{H}_{n}(\epsilon)$, then $\mathbf{t}$ is realized in $\mathcal{H}_{n+2}(\epsilon)$.

Proof.

By construction, $\mathbf{t}$ will be finitely realized in $\mathcal{H}_{n+1}(\epsilon)$, hence realized in $\mathcal{H}_{n+2}(\epsilon)$ by Claim 6.3. ∎

Claim 6.5**.**

If $H$ is an $\epsilon$-hypothesis of $\mathcal{H}_{\infty}(\epsilon)$, then $\mathbf{t}_{H}\in\mathcal{H}_{\infty}(\epsilon)$.

Proof.

Suppose $H=\{(h_{i},\gamma_{i}):i<n\}$. Since it only involves finitely many $h_{i}$’s, let $m<\omega$ be such that $\{h_{i}:i<n\}\subseteq\mathcal{H}_{m}(\epsilon)$. Clearly $H$ remains an $\epsilon$-hypothesis of $\mathcal{H}_{m}$, hence $\mathbf{t}_{H}\in\mathcal{H}_{m+1}(\epsilon)\subseteq\mathcal{H}_{\infty}(\epsilon)$. ∎

Claim 6.6**.**

Assume $\mathcal{H}_{\infty}(\epsilon)$ is Littlestone. If $\mathbf{t}$ is an $\epsilon$-type of $\mathcal{H}_{\infty}(\epsilon)$, then $\mathbf{t}$ is finitely realized in $\mathcal{H}_{\infty}(\epsilon)$, hence realized in $\mathcal{H}_{\infty}(\epsilon)$.

Proof.

By definition of $\epsilon$-type, $\mathbf{t}$ is finitely realized by $\epsilon$-hypotheses in $\mathcal{H}_{\infty}(\epsilon)$. So by Claim 6.5 it is finitely realized in $\mathcal{H}_{\infty}(\epsilon)$. In general, let $g:\mathcal{H}_{\infty}(\epsilon)\rightarrow\{0,1\}$ be any function which is finitely realized in $\mathcal{H}_{\infty}(\epsilon)$. Then Littlestone Minimax (A) is always satisfied (for any $0<\epsilon_{L}<\epsilon_{M}$, in particular for $\epsilon_{M}=\epsilon$, since in condition (A) we can find $h$ which agrees exactly with $g$ on the given finite domain, hence has zero error whatever the weights). So there is an $\epsilon$-hypothesis $H$ of $\mathcal{H}_{\infty}(\epsilon)$ so that $\mathbf{t}_{H}$ represents $g$. By Claim 6.5, $\mathbf{t}_{H}\in\mathcal{H}_{\infty}(\epsilon)$. In particular, $\mathbf{t}$ is realized. ∎

Proof of Theorem 6.2.

Follows from Claim 6.6. ∎

In the next lemma, the intent is that all the classes are over the same set $X$.

Lemma 6.7**.**

Suppose $\mathcal{H}_{\infty}(\epsilon)$ is Littlestone. Suppose $\mathcal{H}\subseteq\mathbf{H}$. If $\mathbf{H}$ is $\epsilon$-saturated then $\mathcal{H}_{\infty}(\epsilon)\subseteq\mathbf{H}$. Hence $\mathcal{H}_{\infty}(\epsilon)$ is the smallest $($under inclusion$)$ $\epsilon$-saturated hypothesis class containing $\mathcal{H}$.111111There can of course be larger $\epsilon$-saturated classes containing a given $\mathcal{H}$, as we can increase the class before taking the saturation.

Proof.

By induction on $n<\omega$ we build an increasing sequence of functions $\langle F_{n}:n<\omega\rangle$ such that:

1. (a)

for $n<\omega$, $F_{n}$ is an injective map from $\mathcal{H}_{n}(\epsilon)$ into $\mathbf{H}$, 2. (b)

for $n<m<\omega$, $F_{m}\upharpoonright\mathcal{H}_{n}(\epsilon)=F_{n}$, and 3. (c)

for $n<\omega$, for every $x\in X$ and $h\in\operatorname{dom}(F_{n})$, $h(x)=1$ if and only if $(F_{n}(h))(x)=1$.

[Since we identify two functions which are the same on $X$, condition (c) amounts to saying the map will be the identity.] For $n=0$, let $F_{0}$ be the identity as $\mathcal{H}\subseteq\mathbf{H}$. Consider $n+1$. Enumerate the elements of $\mathcal{H}_{n+1}(\epsilon)\setminus\mathcal{H}_{n}$ by the appropriate cardinal $\alpha$ as $\langle h_{\beta}:\beta<\alpha\rangle$. The functions $h_{\beta}$ in this list are necessarily distinct functions, and, by definition of $\mathcal{H}_{n+1}$, they are distinct from the elements of $\mathcal{H}_{n}$. It suffices to specify $F_{n+1}$ on each $h_{\beta}$ individually, so fix $h_{\beta}$. By definition of $\mathcal{H}_{n+1}(\epsilon)$, we have $h_{\beta}=\mathbf{t}_{H}$ for some finite weighted $\epsilon$-good set $H=\{(g_{i},\gamma_{i}):i<k\}$ where the $g_{i}$ belong to $\mathcal{H}_{n}(\epsilon)$, but $\mathbf{t}_{H}\notin\mathcal{H}_{n}$. By condition (c), $Y=\{(F_{n}(g_{i}),\gamma_{i}):i<k\}$ is also a weighted $\epsilon$-good set, where the $F_{n}(g_{i})$ belong to $\mathbf{H}$, and $Y$ represents the same function on $X$ as $H$. So $g=\mathbf{t}_{H}=\mathbf{t}_{Y}$ is realized in the class $\{\mathbf{t}_{Y}:Y$ is an $\epsilon$-hypothesis of $\mathbf{H}$ $\}$, hence it is a fortiori finitely realized in this class, hence $g$ is itself a type of $\mathbf{H}$, hence it is realized by some (and indeed exactly one) $f\in\mathbf{H}$ by the assumption of saturation. Note that $f\notin\operatorname{range}(F_{n})$ because we assumed that $\mathbf{t}_{H}\notin\mathcal{H}_{n}$ and $F_{n}$ satisfied condition (c). Set $F_{n+1}(h_{\beta})=f$. ∎

7. Intermediate $\epsilon$

We continue with the assumption that $\mathcal{H}$ is a Littlestone class, of $\operatorname{Ldim}$ $\ell$ and $\operatorname{VC}$ dimension $d$. In 5.6 we saw that if $\epsilon$ is sufficiently small as a function of $d$, then the $\operatorname{VC}$-dimension of $\mathcal{H}_{\infty}$ does not change, and if $\epsilon$ is sufficiently small as a function of $\ell$, then the $\operatorname{Ldim}$ of $\mathcal{H}_{\infty}$ also does not change. In this section we prove the a priori quite surprising theorem that if $\epsilon$ is bounded as a function of $d$ (here we do not refer to $\ell$) then $\mathcal{H}_{\infty}$ remains a Littlestone class possibly of larger $\operatorname{Ldim}$. An upper bound on this larger $\operatorname{Ldim}$ can be obtained from the proof as we will discuss at the end of the section.

In the next theorem, for definiteness we could have said “For any $C\geq 13$” but we give an approximate value, whose calculation is justified in 7.8.

Theorem 7.1**.**

There exists a constant $C\approx 12.0412$ that satisfies the following. Let $\mathcal{H}$ be a Littlestone class, with $\operatorname{Ldim}(\mathcal{H})=\ell$ and $\operatorname{VC}dim(\mathcal{H})=d$. If $0<\epsilon<\frac{1}{Cd}$, then $\mathcal{H}_{\infty}(\epsilon)$ is Littlestone.

More precisely: given $d$, we are free to choose:

1. (1)

any constant $\mathbf{C}\geq C$ 2. (2)

based on $\mathbf{C}$, any $\epsilon_{1}\in(0,1/2)_{\mathbb{R}}$ such that $\mathbf{C}d>20\cdot(d/\epsilon_{1})\cdot\log(1/\epsilon_{1})$ 3. (3)

based on $\epsilon_{1}$, any $\epsilon_{2}\in(0,1/2)_{\mathbb{R}}$ such that $\epsilon_{1}<\epsilon_{2}$ 4. (4)

based on $\epsilon_{2}$, any $\epsilon_{3}\in(0,1/2-\epsilon_{2})_{\mathbb{R}}$

Then, the Littlestone dimension of the class $\mathcal{H}_{\infty}(\epsilon)$ is upper bounded in terms of $d/(\epsilon_{3})^{2}$ and the original Littlestone dimension of $\mathcal{H}$, i.e. $\ell$.

In the statement, “20” in item (2) is an upper bound for the constant in the $\epsilon$-net theorem, and the function bounding the final Littlestone dimension depends on the first part of the proof of Theorem 3 in [2] (bounding Littlestone dimension of fixed finite boolean combinations of elements of a Littlestone class) as discussed at the end of the section.

Before giving the proof, we need to prove several claims. First, we show that the functions in $\mathcal{H}_{\infty}(\epsilon)$ are all $k$-realizable, as long as $k$ is not too large in terms of $\epsilon$. (Compare 5.6 and Example 12.4 below.)

Claim 7.2**.**

Let $k<\lfloor 1/\epsilon\rfloor$. Then $\mathcal{H}_{\infty}(\epsilon)\subseteq\mathcal{H}^{{\operatorname{k-real}}}$.

Proof.

The claim asserts that any $h\in\mathcal{H}_{\infty}(\epsilon)$ is already $k$-realizable in $\mathcal{H}$. If $h\in\mathcal{H}_{\infty}(\epsilon)$ then we can choose a minimal $m$ so that $h\in\mathcal{H}_{m}(\epsilon)$. Clearly $h$ is $k$-realizable in $\mathcal{H}_{m}$, so if $m=0$ we are done. Otherwise, suppose for a contradiction that $m=n+1$ and $h$ is $k$-realizable in $\mathcal{H}_{n+1}$ but not in $\mathcal{H}_{n}$.

As $h\in\mathcal{H}_{n+1}$ there is a weighted $\epsilon$-good set $H=\{(g_{i},\gamma_{i}):i<s\}$ which represents $h$, hence for each $x\in X$, $\sum\{\gamma_{i}:i<s,g_{i}(x)\neq h(x)\}<\epsilon$. Hence given any $x_{0},\dots,x_{k-1}$ from $X$, we have that $\sum\{\gamma_{i}:i<s,j<k,g_{i}(x_{j})\neq h(x_{j})\}<\epsilon k\leq 1$ (by choice of $\epsilon$). So there exists $(g,\gamma)\in H$ such that $g(x_{j})=h(x_{j})$ for $j=0,\dots,k-1$. By definition $g\in\mathcal{H}_{n}$. This shows $h$ is $k$-realizable in $\mathcal{H}_{n}$, contradicting our choice of $n$. ∎

In the following claim, we fix $C$ and $\epsilon$ from Theorem 7.1. Note that this background $\epsilon$ is not used in this claim.

The claim analyzes a single function $f$ from $\mathcal{H}^{\operatorname{k-real}}$ in the case where $k=Cd$. Recall that a priori we do not know the complexity of $\mathcal{H}^{{\operatorname{Cd-real}}}$, but we do know that $\mathcal{H}$ is Littlestone and indeed of $\operatorname{VC}$ dimension $d$. Let $\mathcal{H}_{f}$ denote the class of symmetric differences of $f$ with each element of $\mathcal{H}$, which remains a $\operatorname{VC}$ class of dimension $d$. Given any measure (we will only use finitely supported probability measures) $\mu$ on $X$ and $h_{f}\in\mathcal{H}$, let $\mu(h_{f})$ mean $\mu(\{x\in X:h_{f}(x)=1\})$. There are two key ideas. First, since $\mathcal{H}_{f}$ is a $\operatorname{VC}$ class, the $\epsilon$-net theorem [note: we should call it the $\epsilon_{1}$-net theorem since we are applying it to an $\epsilon_{1}$ we shall choose presently and not our background $\epsilon$] applies: for some $M=M(\epsilon_{1},d)$, for any measure $\mu$ on $X$ there is $S\subseteq X$, $|S|\leq M$ which “sees” all nontrivial hypotheses (i.e. for every $h_{f}$ of measure $\geq\epsilon_{1}$ there exists $s\in S$ so that $h_{f}(s)=1$). Since $\mathcal{H}_{f}$ measures symmetric difference with $f$, this amounts to saying that if $h\in\mathcal{H}$ differs from $f$ by more than $\epsilon_{1}$, the set $S$ sees it. Second, note this along with Claim 7.2 tells us that for any choice of $\epsilon_{1}$ satisfying Theorem 7.1(2), since $Cd>M(\epsilon_{1},d)$, when $\mu$ is any finitely supported probability measure on $X$ and $S$ is an $\epsilon_{1}$-net, $Cd>|S|$ hence the assumption that $f$ is $Cd$-realizable in $\mathcal{H}$ means that there is some $h\in\mathcal{H}$ so that $h$ and $f$ agree on $S$, hence the global difference between $h$ and $f$ is $<\epsilon_{1}$.

Claim 7.3**.**

Let $\mathcal{D}$ be a distribution over $X$ with finite support, and let $\mathcal{H}$ be [a Littlestone class] of $\operatorname{VC}dim(\mathcal{H})=d$. Let $C$ be as in Theorem 7.1. If $\epsilon_{1}\in(0,1/2)_{\mathbb{R}}$ satisfies the inequality $C>20\cdot\epsilon_{1}^{-1}\cdot\log(\epsilon^{-1}_{1})$ then for every $f\in\mathcal{H}^{{\operatorname{Cd-real}}}$, there exists $h\in\mathcal{H}$ such that

[TABLE]

Proof.

Define $\mathcal{H}_{f}:=\{h_{f}:h\in\mathcal{H}\}$, where for each $h\in\mathcal{H}$ and $x\in X$ we define $h_{f}(x):=\delta_{f(x)\neq h(x)}$, as the symmetric difference of $h$ and $f$. Since the map $h\mapsto h_{f}$ is an involution, shattered sets are the same for both classes, and $VCdim(\mathcal{H})=VCdim(\mathcal{H}_{f})=d$.

Then, by $\epsilon$-net Theorem on $\mathcal{H}_{f}$ (see e.g. Theorem 10.2.4 in [22]), there exists a constant $C_{net}<20$, and a number $M:=m(d,\epsilon_{1})\leq C_{\text{net}}\cdot d\cdot(1/\epsilon_{1})\cdot\log(1/\epsilon_{1})$ such that a sample $\mathcal{S}$ of size $M$ from $X\sim\mathcal{D}$, satisfies with positive probability that:

[TABLE]

Following the correspondence between $\mathcal{H}$ and $\mathcal{H}_{f}$, the above is equivalent to

[TABLE]

The condition on $\epsilon_{1}$ implies $M\leq Cd$. Let $\mathcal{S}$ be a sample with the above property. Since $|\mathcal{S}|\leq M\leq Cd$, then $f\in\mathcal{H}^{{\operatorname{Cd-real}}}$ implies there is some $h\in\mathcal{H}$ such that for all $x\in\mathcal{S}$, $f(x)=h(x)$. By contrapositive, then, $Pr_{x\sim\mathcal{D}}[h(x)\neq f(x)]<\epsilon_{1}$. ∎

We continue to analyze a given $f\in\mathcal{H}^{{\operatorname{Cd-real}}}$. The next Claim applies Littlestone minimax to conclude that any such $f$ can be represented by a weighted $\epsilon_{2}$-majority vote of elements of $\mathcal{H}$, for any $\epsilon_{2}\in(\epsilon_{1},1/2)_{\mathbb{R}}$. Why? Condition (A) of the Littlestone Minimax, Theorem A above, is exactly what is guaranteed by Claim 7.3.

Claim 7.4**.**

Let $\mathcal{D},\mathcal{H},d,C,\epsilon_{1}$ be defined as before. In particular, recall that $\mathcal{H}$ is a Littlestone class with $VCdim(\mathcal{H})=d$. Fix $f\in\mathcal{H}^{{\operatorname{Cd-real}}}$. For any $\epsilon_{2}\in(0,1/2)$ such that $\epsilon_{2}>\epsilon_{1}$, there exists a finite weighted sequence $\{(h_{i},\gamma_{i})\;:\;i<n\}$ in $\mathcal{H}$ such that, for every $x\in X$:

1. (i)

there is an $\epsilon_{2}$-majority vote $\mathbf{t}(x)\in\{0,1\}$, i.e. a value for which

[TABLE] 2. (ii)

$\mathbf{t}(x)=f(x)$.

Proof.

Suppose we are given $\{(x_{t},w_{t}):t<s\}$ as in condition (A) of Littlestone Minimax. By 7.3 there is some $h\in\mathcal{H}$ whose weighted error against $f$ is $<\epsilon_{1}$. So by condition (B) of Littlestone Minimax there is an $\epsilon_{2}$-hypothesis $H=\{h_{i},\gamma_{i}:i<n\}$ of $H$ so that $\mathbf{t}_{H}=f$. ∎

We have seen that our $f\in\mathcal{H}^{{\operatorname{Cd-real}}}$ is expressible as a weighted $\epsilon_{2}$-majority vote of some $H\subseteq\mathcal{H}^{{\operatorname{Cd-real}}}$. $H$ is finite, but a priori we have no bound on its size. Our last claim shows that we may assume a uniform finite bound on the size of $H$, as a function of $d$ and any fixed $\epsilon_{3}\in(0,1/2-\epsilon_{2})_{\mathbb{R}}$. (Why? Use the fact that the dual of any VC class is also a VC class, take the finitely supported measure on $\mathcal{H}$ corresponding to the weighted $\epsilon_{2}$-good set $H$, and choose a representative sample from it of size about $d/(\epsilon_{3})^{2}$, as the proof describes.)

Definition 7.5**.**

Given a hypothesis class $\mathcal{H}$ and an odd $p\in\mathbb{N}$, let $\langle g_{j}:j<p\rangle$ be a sequence of elements of $\mathcal{H}$, possibly with repetition. The majority vote of this sequence is defined to be the function $g:X\rightarrow\{0,1\}$ given by: $g(x)=1$ if $|\{j<p:g_{j}(x)=1\}|>(p-1)/2$ and $g(x)=0$ otherwise. Let

[TABLE]

be the set of functions arising as majority votes of sequences $\langle g_{j}:j<p\rangle$ in the $p$-fold Cartesian power $\mathcal{H}\times\cdots\times\mathcal{H}$.

In particular, definition of majority in 7.5 is the counting majority and is not directly related to a background epsilon.

Claim 7.6**.**

Let $\mathcal{D},\mathcal{H},d,C,\epsilon_{2}$ be defined as before. For some odd $p\in\mathbb{N}$, for every $f\in\mathcal{H}^{{\operatorname{Cd-real}}}$, there exists a sequence $\langle f_{i}:i<p\rangle$ of elements of $\mathcal{H}$, possibly with repetition, whose majority vote is $f$. In other words, $\mathcal{H}^{{\operatorname{Cd-real}}}\subseteq\operatorname{Maj}_{p}(\mathcal{H})$.

Proof.

Let $f\in\mathcal{H}^{{\operatorname{Cd-real}}}$ be given, and let $H=\{(h_{i},\gamma_{i}):i<n\}$ be the corresponding $\epsilon_{2}$-hypothesis given by 7.4. Recall that $\mathcal{H}_{*}:=\operatorname{dual}(\mathcal{H})$ (3.16) is also a VC class, of VC dimension $d_{*}\leq 2^{d+1}-1$. So it satisfies uniform convergence: namely, there is a constant $c_{*}=c_{*}(\epsilon_{3},d_{*})$ and a sample size $m_{*}=m_{*}(\epsilon_{3},d_{*})\leq c_{*}d_{*}(\epsilon_{3})^{-2}$ with the following property. For any probability measure on $\mathcal{H}$ [recall that in the dual $\mathcal{H}_{*}$, the domain is $\mathcal{H}$], and in particular for that corresponding to $H=\{(h_{i},\gamma_{i}):i<m\}$, for any $p\geq m_{*}$ [for definiteness, for us, let $p$ be the smallest odd number $\geq m_{*}$], with high probability an iid sample $\langle g_{t}:t<p\rangle$ satisfies: for every $x\in X$, the “empirical and actual measures are close,” meaning that

[TABLE]

(Informally, each $x\in\mathcal{H}_{*}$ determines the set of $h\in\mathcal{H}$ which label it by 1; the size of that set according to $H$ should be close to the size of that set according to the sample, which is of course sub-sampled from $\{h_{i}:i<n\}$ by our choice of measure.)

Since this happens with high probability, in particular we can fix some sequence $\langle g_{t}:t<p\rangle$ on which this happens. Now $\sum\{\gamma_{i}:i<n,h_{i}(x)=1\}$ is either $<\epsilon_{2}$ or $\geq 1-\epsilon_{2}$, so $\frac{|\{t<p:g_{t}(x)=1\}|}{|p|}$ is either $<\epsilon_{2}+\epsilon_{3}$ or $\geq 1-\epsilon_{2}-\epsilon_{3}$. In other words, since $(\epsilon_{2}+\epsilon_{3})<1/2$, we have that for every $x$, $\langle g_{t}(x):t<p\rangle$ has a numerical majority which agrees with $H$ and hence represents $f$. ∎

Now we can conclude by applying known bounds on the $\operatorname{Ldim}$ of fixed finite boolean combinations of elements of a Littlestone class. We will use:

Fact 7.7** ([2], Theorem 3).**

If $\mathcal{H}$ is Littlestone then for any fixed odd $p\in\mathbb{N}$, $\operatorname{Maj}_{p}(\mathcal{H})$ is also Littlestone. Concretely, we have the following upper bound:

[TABLE]

The proof of this fact uses a technique named Online boosting. In model theoretic language, the familiar fact is that if $\varphi$ is a stable formula, then any fixed finite boolean combination of $\varphi$’s is also stable, though of a possibly larger rank.

We now formally assemble the pieces to prove Theorem 7.1:

Proof of Theorem 7.1.

By Observation 7.2, regardless of what our constant $c$ is, $\epsilon<1/cd$ implies that $\mathcal{H}_{\infty}(\epsilon)\subseteq\mathcal{H}^{{\operatorname{cd-real}}}$. Also, as long as $cd>d+1$, Theorem 5.6(4), $\operatorname{VC}(\mathcal{H}_{\infty}(\epsilon))=\operatorname{VC}(\mathcal{H})=d<\infty$.

For the rest of the proof, fix $C$ satisfying the statement of the theorem (recall $C\approx 12.0412$) and choose some $\epsilon_{1}>0$ satisfying $C>20\cdot\epsilon_{1}^{-1}\cdot\log(\epsilon^{-1}_{1})$. Then Claim 7.3 will allow us to find, for every $f\in\mathcal{H}^{{\operatorname{Cd-real}}}$, some $h\in\mathcal{H}$ with error less than $\epsilon_{1}$ with respect to $f$, by realizability and the $\epsilon$-net property. For any $\epsilon_{2}>\epsilon_{1}$, Claim 7.4 yields an $\epsilon_{2}$-hypothesis $H$ on $\mathcal{H}$ such that $\mathbf{t}_{H}$ equals $f(x)$, for all $x\in X$. Then, Claim 7.6 allows us to find a uniform size for these representations: it shows there is $p\in\mathbb{N}$ so that, sampling $p$ times from the distribution given by $H$, and then taking the $p$-wise majority vote, we obtain $f(x)$ for all $x\in X$.

We have shown that under the assumptions of the theorem

[TABLE]

Concluding by Fact 7.7, $\mathcal{H}_{\infty}(\epsilon)$ is Littlestone since $\operatorname{Maj}_{p}(\mathcal{H})$ is. ∎

Remark 7.8**.**

The choice $C\approx 12.0412$ comes from the following considerations:

• •

The decreasing and injective behavior of $f(x)=\frac{1}{x}\cdot\log(\frac{1}{x})$ over $(0,1/2)_{\mathbb{R}}$.

• •

The argument in Claim 7.3 needs the inequality $Cd>d\cdot f(\epsilon_{1})$ to hold

• •

Continuity of $f(x)$ at $1/2$, and $12.0412\approx\lim_{x\to 1/2^{-}}f(x)$.

8. Conclusions for Littlestone classes

In this section, we summarize the main results of this paper so far into the following theorem, also visible in Figure 1.

Theorem 8.1**.**

Let $(X,\mathcal{H})$ be a hypothesis class such that $\operatorname{Ldim}(\mathcal{H})=\ell$ and $\operatorname{VC}(\mathcal{H})=d$. The saturations of these classes satisfy the following:

1. (a)

When $\epsilon\leq\frac{1}{\ell+1}$, $\operatorname{Ldim}(\mathcal{H}_{\infty}(\epsilon))=\operatorname{Ldim}(\mathcal{H})=\ell$; 2. (b)

When $\epsilon\leq\frac{1}{d+1}$, $\operatorname{VC}(\mathcal{H}_{\infty}(\epsilon))=\operatorname{VC}(\mathcal{H})=d$. 3. (c)

For $\epsilon<\frac{1}{Cd}$, for some $C\gtrapprox 12.0412$, $\operatorname{Ldim}(\mathcal{H}_{\infty}(\epsilon))<\infty$ although it need not stay the same as $\operatorname{Ldim}(\mathcal{H})$; 4. (d)

For $\epsilon>\frac{1}{d+1}$, it is possible for both $\operatorname{Ldim}(\mathcal{H}_{\infty})$ and $\operatorname{VC}(\mathcal{H}_{\infty})$ to be infinite.

Proof.

Parts $(a)$ and $(b)$ are both shown in Theorem 5.6. Part $(c)$ is shown in Theorem 7.1. Part $(d)$ is shown by Example 12.1. ∎

9. Definition of $\epsilon$-saturation for $\mathcal{G}$

We now give the definitions in the case of graphs. A crucial role will be played by excellent sets. Remember these can be different from good sets; see §12.2 .

Convention 9.1**.**

When dealing with graphs we will always assume $\epsilon<1/4$.

Convention 9.2**.**

$\mathcal{G}=(V,E)$, which we may abbreviate $\mathcal{G}$, will usually denote a stable graph with:

• •

finite edge stability, also called threshold dimension, $s$,

• •

hence finite Littlestone dimension $\ell$,

• •

hence finite VC dimension $d$.

The vertex set $V$ can be finite, but need not be. We allow self-loops, that is, we allow $(v,v)$ as an edge for any vertex $v$.121212Briefly, this is because we will add $\epsilon$-excellent sets of vertices as new “virtual” vertices and any such set $A$ will have a well-defined edge or nonedge to any other such set, including itself. There is a priori no reason to erase the information of which of the two it is.

As before, we will repeat the assumption of stability in the main definitions and theorems which use it. We will often write “$v\in\mathcal{G}$” to mean $v\in V(\mathcal{G})$, and “$v\sim w$” to mean $(v,w)\in E(\mathcal{G})$.

Compared to the definition of $\epsilon$-hypothesis, the definition of $\epsilon$-vertex will require an extra step due to the fact that we would like edges between $\epsilon$-vertices to be well defined (this corresponds to $\epsilon$-excellent in the stable regularity lemma). As before, we will use weighted good and excellent sets.

Definition 9.3**.**

Given $\epsilon$ and $\mathcal{G}$, a weighted $\epsilon$-good set of $\mathcal{G}$ is a finite set of pairs:

[TABLE]

where $k\in\mathbb{N}$, and:

1. (1)

$\{v_{i}:i<k\}\subseteq\mathcal{G}$, and $\langle v_{i}:i<k\rangle$ is without repetition. 2. (2)

each $\gamma_{i}\in(0,1]_{\mathbb{R}}$, and $\sum\{\gamma_{i}:i<k\}=1$. 3. (3)

($$\epsilon-goodness$)$* For each $v\in V$, either*

[TABLE]

In the first case, write $\mathbf{t}(A,v)=1$, and in the second $\mathbf{t}(A,v)=0$.

Definition 9.4**.**

Given $\epsilon$ and $\mathcal{G}$, an $\epsilon$-vertex $($or a weighted $\epsilon$-excellent set$)$ of $\mathcal{G}$ is a finite set of pairs:

[TABLE]

where $k\in\mathbb{N}$, $B$ is a weighted $\epsilon$-good set, and moreover:

1. (4)

($$\epsilon-excellence$)$* For every weighted $\epsilon$-good set $A$, either*

[TABLE]

In the first case, write $\mathbf{t}(B,A)=1$, and in the second $\mathbf{t}(B,A)=0$.

That 9.4 is sufficiently symmetric for our purposes is justified by the following (which also explains our convention that in working with graphs, $\epsilon<1/4<2-\sqrt{3}$).

Remark 9.5**.**

Let $\epsilon\in(0,2-\sqrt{3})$, and let $A$ and $B$ be two $\epsilon$-excellent weighted sets of $G$. Then, the values $\mathbf{t}(A,B)$ and $\mathbf{t}(B,A)$ are well-defined and equal.

Proof.

Suppose for a contradiction that $\mathbf{t}(A,B)=1$ and $\mathbf{t}(B,A)=0$. This implies that, if the weight for each $a_{i}$ is $\gamma_{i}$ and the weight for each $b_{i}$ is $\rho_{i}$, then:

[TABLE]

The first bound above implies that the total sum of the products $\gamma_{i}\rho_{j}$ over all edges from $A$ to $B$ is lower bounded by

[TABLE]

The second bound above implies that the same total sum of products of weights described above is upper bounded by

[TABLE]

The symmetry of edges in $G$ and these two bounds imply that

[TABLE]

These values for $\epsilon$ are ruled out by the hypothesis that $\epsilon<2-\sqrt{3}$. ∎

Definition 9.6**.**

Given $\epsilon$ and $\mathcal{G}$, whenever $A,B$ are $\epsilon$-vertices, say there is an $\epsilon$-edge between $A,B$, in symbols $A\sim B$ or $A\sim_{\epsilon}B$, to mean that $\mathbf{t}(A,B)=\mathbf{t}(B,A)=1$, and say there is an $\epsilon$-nonedge, in symbols $A\not\sim B$ or $A\not\sim_{\epsilon}B$, to mean that $\mathbf{t}(A,B)=\mathbf{t}(B,A)=0$.

Notation 9.7**.**

Whenever $A$ is an $\epsilon$-vertex, let $\mathbf{t}_{A}$ be the function whose domain is the set of $\epsilon$-vertices of $\mathcal{G}$ and whose range is $\{0,1\}$, given by $B\mapsto\mathbf{t}(A,B)=\mathbf{t}(B,A)$.

Convention 9.8**.**

Since for any $v\in V$, $\{v\}$ is an $\epsilon$-vertex, we may drop parentheses, writing $v$ for $\{v\}$ or $w$ for $\{w\}$ in expressions like $\mathbf{t}$ or $\sim_{\epsilon}$.

To define the $\epsilon$-saturation, we will formally extend our graphs by adding $\epsilon$-vertices, and so all our edges are a priori $\epsilon$-edges. Note, however, that in Definition 4.9 in some sense we constantly forget this: the graphs $\mathcal{G}_{n}$ at each stage in the construction are actual graphs (which come from formally adding $\epsilon$-vertices as actual new vertices, then “rounding up” $\sim_{\epsilon}$ to $\sim$ and “rounding down” $\not\sim_{\epsilon}$ to $\not\sim$). Since there will often be many $\epsilon$-vertices with the same $\sim_{\epsilon}$-neighborhood, in order to avoid triviality, we only add one representative of each such neighborhood class.131313 Indeed, since $\mathbf{t}(A,B)=\mathbf{t}(A^{\prime},B^{\prime})$ whenever $A,A^{\prime}$ and $B,B^{\prime}$ have the same $\sim_{\epsilon}$-neighborhoods, when we will want to remember these arose from $\epsilon$-vertices, we are free to calculate by choosing some possibly different representative of the class. This is the analogue of not repeating functions in the case of $\mathcal{H}$ above.

Definition 9.9** ($\epsilon$-types).**

Given $\mathcal{G}=(V,E)$ and a function $\mathbf{t}:V\rightarrow\{0,1\}$:

1. (1)

$\mathbf{t}$* is $k$-realized in $\mathcal{G}$, for some given $k\in\mathbb{N}$, if for any $W\subseteq V$, $|W|\leq k$ there is $v\in V$ such that $\mathbf{t}_{\{(v,1)\}}\upharpoonright W=\mathbf{t}\upharpoonright W$.* 2. (2)

$\mathbf{t}$* is finitely realized in $\mathcal{G}$ if it is $k$-realized in $\mathcal{G}$ for all $k\in\mathbb{N}$.* 3. (3)

Given also $\epsilon<1/4$, an $\epsilon$-type for $\mathcal{G}$ is a function $\mathbf{t}:V\rightarrow\{0,1\}$ such that for any finite $W\subseteq V$, there is an $\epsilon$-vertex $A$ of $\mathcal{G}$ such that141414informally, we could say the function is finitely realized by \{\mathbf{t}_{A}:A\mbox{ an \epsilon$-vertex of$\mathcal{G}}\}. **

[TABLE]

The central definition of this paper, in the case of graphs, is:

Definition 9.10** (The $\epsilon$-saturation of $\mathcal{G}$).**

Suppose we are given a graph $\mathcal{G}=(V,E)$ and $0<\epsilon<1/4$. Define by induction on $n\leq\omega$ a graph $\mathcal{G}_{n}(\epsilon)=(V_{n},E_{n})$:

1. (1)

$\mathcal{G}_{0}(\epsilon)=\mathcal{G}=(V,E)$. 2. (2)

$\mathcal{G}_{n+1}(\epsilon)=(V_{n+1},E_{n+1})$* where:151515Why do we not ask that for every $\epsilon$-vertex $A$ of $V_{n}$, there is precisely one $\epsilon$-vertex $B$ of $V_{n}$ such that $B\in V_{n}\cup W$ and $\mathbf{t}_{B}=\mathbf{t}_{A}$? This would impose a retroactive restriction on $G_{n}$ saying that there cannot be two singletons with the same neighborhood. There is a priori no reason this should be true of $G_{0}$, that is, we do not require $G_{0}$ to be prime, even though we take care not to add instances of non-primality later on.*

1. (a)

$V_{n+1}=V_{n}\cup W$* where $W$ is a representative set of $\epsilon$-vertices of $V_{n}$ meaning: for every $\epsilon$-vertex $A$ of $V_{n}$ with $|A|>1$, there is precisely one $\epsilon$-vertex $B$ of $V_{n}$ such that $B\in W$ and $\mathbf{t}_{B}=\mathbf{t}_{A}$.* 2. (b)

$E_{n+1}$* is given by:*

1. (i)

for $(v,w)\in V_{n}\times V_{n}$, $(v,w)\in E_{n+1}$ iff $(v,w)\in E_{n}$; 2. (ii)

*otherwise, for $(v,w)\in(V_{n+1}\times V_{n+1})\setminus(V_{n}\times V_{n})$, *

$(v,w)\in E_{n+1}$* iff $v\sim_{\epsilon}w$.* 3. (3)

$\mathcal{G}_{\infty}(\epsilon):=\mathcal{H}_{\omega}(\epsilon)=(V_{\infty},E_{\infty})$* where $V_{\infty}=V_{\omega}=\bigcup\{V_{n}:n<\omega\}$ and $E_{\infty}=E_{\omega}=\bigcup\{E_{n}:n<\omega\}$.*

We call $\mathcal{G}_{\infty}(\epsilon)$ “the $\epsilon$-saturation of $\mathcal{G}$.”

Here also the name, and the connection to $\epsilon$-types, will be justified in due course. Also, we could have continued the definition by transfinite induction, but this is not used here.

10. Characterization of $\epsilon$-saturation for stable graphs

The following definition and lemma have a natural analogue for hypotheses classes, which we do not state separately. Note that in the case of a hypothesis class $(X,\mathcal{H})$, the domain $X$ did not change, we simply added new functions; whereas here the domain is a priori constantly changing, so things have a slightly different character.

Definition 10.1**.**

A partial $\epsilon$-type of a graph or hypothesis class means that we allow the function $\mathbf{t}$ to be partial.

Lemma 10.2** (Types can be extended).**

Let $\mathbf{s}$ be a partial $\epsilon$-type of a graph $G$. Then $\mathbf{s}$ can be extended to an $\epsilon$-type, that is, there is an $\epsilon$-type $\mathbf{t}\supseteq\mathbf{s}$. In fact, $\mathbf{s}$ can be extended to $V\cup\{B:B$ is a weighted $\epsilon$-good subset of $G$ $\}$.

Proof.

It will be notationally convenient to identify functions with their graphs. Enumerate the elements of

[TABLE]

by the cardinal $\kappa\geq|G|$) as

[TABLE]

Let us show by induction on $\alpha<\kappa$ that we can build an increasing continuous chain of partial $\epsilon$-types $\mathbf{t}_{\alpha}$ such that:

1. (1)

$\mathbf{t}_{0}=\mathbf{s}$ 2. (2)

each $\mathbf{t}_{\alpha}$ is an $\epsilon$-type, i.e. finitely realized by $\epsilon$-vertices 3. (3)

$\beta<\alpha\implies\mathbf{t}_{\beta}\subseteq\mathbf{t}_{\alpha}$ 4. (4)

if $\alpha$ is a limit ordinal then $\mathbf{t}_{\alpha}=\bigcup\{\mathbf{t}_{\beta}:\beta<\alpha\}$ 5. (5)

if $\alpha=\beta+1$ then $c_{\beta}\in\operatorname{dom}(\mathbf{t}_{\alpha})$.

For $\alpha=0$ and $\alpha$ limit this is easy as being finitely realized by $\epsilon$-vertices is preserved under unions of chains.

Let $\alpha=\beta+1$. If $c_{\beta}\in\operatorname{dom}(\mathbf{t}_{\beta})$, set $\mathbf{t}_{\alpha}=\mathbf{t}_{\beta}$ and we are done. Otherwise, we ask if $\mathbf{t}_{\beta}\cup\{(c_{\beta},1)\}$ is finitely realizable by $\epsilon$-vertices. If not, there is a finite $\mathbf{t}^{\prime}\subseteq\mathbf{t}_{\beta}$ so that $\mathbf{t}^{\prime}\cup\{(c_{\beta},1)\}$ is not realized by any $\epsilon$-vertex of $\mathcal{G}$. Likewise if $\mathbf{t}_{\beta}\cup\{(c_{\beta},0)\}$ is not finitely realizable by $\epsilon$-vertices, there is a finite $\mathbf{t}^{\prime\prime}\subseteq\mathbf{t}_{\beta}$ such that $\mathbf{t}^{\prime\prime}\cup\{(c_{\beta},0)\}$ is not finitely realizable by $\epsilon$-vertices. But then $\mathbf{t}^{\prime}\cup\mathbf{t}^{\prime\prime}$ cannot be finitely realized by $\epsilon$-vertices, because if it were, say by $A$, this same $A$ would have to take some value on $c_{\beta}$ which would contradict one of our two assumptions. So one of the two additions must be possible, completing this step and hence the proof. ∎

Theorem 10.3**.**

Let $\mathcal{G}$ be a $k$-stable graph, $\epsilon<1/4$ and $\mathcal{G}_{\infty}(\epsilon)$ its saturation. Suppose $\mathcal{G}_{\infty}(\epsilon)$ is also Littlestone. Then $\mathcal{G}_{\infty}(\epsilon)$ realizes all of its $\epsilon$-types.

The proof will depend on several claims, slightly different than Theorem 6.2.

Claim 10.4**.**

If $V=\{(v_{i},\gamma_{i}):i<m\}$ is an $\epsilon$-vertex of $G_{\infty}(\epsilon)$ then it is also an $\epsilon$-vertex of $G_{n}(\epsilon)$ for some $n<\omega$.

Proof.

Let $n<\omega$ be such that $\{v_{i}:i<m\}\subseteq V_{n}$. So $V$ is a weighted set of elements of $G_{n}(\epsilon)$. Considered as a weighted set of elements of $G_{\infty}(\epsilon)$, $V$ is good, and indeed excellent. We need to check that both remain true in $G_{n}(\epsilon)$. Goodness is easy: for any $x\in G_{\infty}(\epsilon)$, $\sum\{\gamma_{i}:\mathbf{t}(x,V)=1\}$ is either $<\epsilon$ or $\geq 1-\epsilon$, and so a fortiori this is true for the $x\in G_{n}(\epsilon)$. In other words:

($\star$) if $A$ is a weighted set of elements of $G_{n}(\epsilon)$, and $A$ is a weighted $\epsilon$-good set in $G_{\infty}(\epsilon)$, then also $A$ is a weighted $\epsilon$-good set in $G_{n}(\epsilon)$.

The subtlety about excellence is that we need to quantify over all $\epsilon$-good weighted sets of the given graph, and a priori those may be different in $G_{n}(\epsilon)$. We show they are in fact the same. Suppose $B$ were an $\epsilon$-good weighted set in $G_{n}(\epsilon)$ which ceased to be good in $G_{n+1}(\epsilon)$. (Since we made no assumptions about the minimality of $n$ this is the general case.) This would mean that we added some new vertex which split $B$ in an balanced way. But the vertices we add are all $\epsilon$-vertices, that is, they arise as weighted $\epsilon$-excellent sets of $G_{n}(\epsilon)$. In particular they have $\epsilon$-majority opinions on any weighted $\epsilon$-good set. We have shown that:

$(\star\star)$ If $C$ is a weighted set of $G_{n}(\epsilon)$, then $C$ is an $\epsilon$-good weighted set of $G_{\infty}(\epsilon)$ if and only if it is an $\epsilon$-good weighted set of $G_{n}(\epsilon)$.

So $V$ is a weighted $\epsilon$-excellent set of $G_{n}(\epsilon)$, i.e. an $\epsilon$-vertex of $G_{n}(\epsilon)$. ∎

Corollary 10.5**.**

If $v$ is an $\epsilon$-vertex of $\mathcal{G}_{\infty}(\epsilon)$ then $v\in V_{\infty}$.

Corollary 10.6**.**

If $\mathbf{t}$ is an $\epsilon$-type of $\mathcal{G}_{\infty}(\epsilon)$ then $\mathbf{t}$ is finitely realized in $\mathcal{G}_{\infty}(\epsilon)$.

The subtlety of the next claim is that simply applying Littlestone Minimax to an $\epsilon$-type returns some weighted $\epsilon$-good $A$, but Claim 10.6 only tells us that weighted $\epsilon$-excellent sets belong to $B$.

Claim 10.7**.**

If $\mathbf{t}$ is finitely realized in $\mathcal{G}_{\infty}(\epsilon)$ then $\mathbf{t}$ is realized in $\mathcal{G}_{\infty}(\epsilon)$.

Proof.

Recall that $V_{\infty}$ denotes the vertex set of $\mathcal{G}_{\infty}(\epsilon)$. Let $\mathbf{t}$ be an $\epsilon$-type of $\mathcal{G}_{\infty}(\epsilon)$ and by 10.2, let $\mathbf{t}_{*}$ be an extension to the domain

[TABLE]

[Recall that weighted $\epsilon$-good sets are always finite.] Since by construction $\mathbf{t}_{*}$ is finitely realized by $\epsilon$-vertices of $\mathcal{G}_{\infty}(\epsilon)$, by 10.5, $\mathbf{t}_{*}$ is finitely realized in $\mathcal{G}_{\infty}(\epsilon)$. Hence by Littlestone Minimax $\mathbf{t}_{*}$ is realized by some weighted $\epsilon$-good $A$. But now by construction $A$ has an opinion about every weighted $\epsilon$-good $B$ of $\mathcal{G}_{\infty}(\epsilon)$. Hence $A$ is a weighted $\epsilon$-excellent set, in other words an $\epsilon$-vertex, which realizes the type. Invoke 10.5 one more time to conclude $A\in V_{\infty}$, which completes the proof. ∎

We can now complete the proof of the theorem:

Proof of 10.3.

By Claims 10.6 and 10.7. ∎

Theorem 5.6 goes through for graphs with a similar proof: we need to make the natural updates to use “excellent” rather than “good” and to deal with the fact that things are no longer bipartite.

Lemma 10.8**.**

Let $s\in\mathbb{N}$, $s\geq 2$, and $\epsilon<1/2^{s}$. Let $\langle a_{\eta}:\eta\in{{}^{s>}2}\rangle$ and $\langle b_{\rho}:\rho\in{{}^{s}2}\rangle$ be a special tree of height $s$ in $G_{\infty}(\epsilon)$. Then there is also a special tree of height $s$ in $G$.

Proof.

Let $m<\omega$ be minimal so that $\{a_{\eta}:\eta\in{{}^{s>}2}\}\cup\{b_{\rho}:\rho\in{{}^{s}2}\}\subseteq G_{m}(\epsilon)$; if $m=0$ we are done, so assume $m=n+1$ for some $n\in\mathbb{N}$. For each $\eta\in{{}^{s>}2}$ choose a corresponding $\epsilon$-vertex $A_{\eta}$ of $G_{n}$ and for each $\rho\in{{}^{\rho}2}$ choose a corresponding $\epsilon$-vertex $B_{\rho}$ of $G_{n}$, so for $\eta\trianglelefteq\rho$, $\mathbf{t}(A_{\eta},B_{\rho})=1$ if and only if there is an edge between $a_{\eta}$ and $b_{\rho}$.

Since these are virtual vertices and virtual edges, there are two steps. First, for each $\rho\in{{}^{s}2}$, define an error set $E_{\rho}=\{(b_{i},\gamma_{i})\in B_{\rho}:\mathbf{t}(b_{i},A_{\eta})\neq\mathbf{t}(B_{\rho},A_{\eta})\}$ which will have weight $<\epsilon s$. So there is $(b,\gamma)\in B_{\rho}\setminus E_{\rho}$ and we can set $b^{\prime}_{\rho}:=b$. Second, now we are in the setting of having a special tree in $G_{n}$ with internal nodes $\langle A_{\eta}:\eta\in{{}^{s>}2}\rangle$ which may be virutal, and leaves $\langle b^{\prime}_{\rho}:\rho\in{{}^{s}2}\rangle$ which are vertices of $G_{n}$, so as in 5.6(2) we find a special tree of height $s$, $\langle a^{\prime}_{\eta}:\eta\in{{}^{s>}2}\rangle$, $\langle b^{\prime}_{\rho}:\rho\in{{}^{s}2}\rangle$ in the vertices of $G_{n}$, contradicting the choice of $n$. ∎

Conclusion 10.9**.**

Let $\mathcal{G}$ be a graph.

1. (1)

If $\operatorname{Ldim}(\mathcal{G}):=\ell<\infty$, and $\epsilon<1/2^{\ell+1}$, then

[TABLE] 2. (2)

If in addition $\epsilon<1/2^{\ell+2}$, then

[TABLE] 3. (3)

If $\operatorname{VC}(\mathcal{G}):=d<\infty$, and $\epsilon<1/2^{d+1}$, then

[TABLE]

Proof.

To prove this, we can use a parallel series of arguments as in 5.6, to transfer a full special tree, half graph, or VC-tree, from $\mathcal{G}_{n+1}$ to $\mathcal{G}_{n}$, and inductively, all the way back to $\mathcal{G}$. ∎

We have seen that $\epsilon$-saturation for graphs behaves very well for small $\epsilon$. What about intermediate $\epsilon$? Can it still happen that bounding $\epsilon$ in terms of the $VC$ dimension gives us some control over the Littlestone dimension of $\mathcal{G}_{\infty}(\epsilon)$?

Remark 10.10**.**

In considering extending the arguments of §7 to graphs, proving an analogue of Claim 7.2 doesn’t make sense globally since the domain of a function $($that is, the neighborhood of a vertex$)$ in $G_{\infty}$ includes many vertices which aren’t in $G$, hence $k$-realizability doesn’t make sense out of the box. We leave it as an interesting open question whether it is possible to conclude that $\mathcal{G}_{\infty}(\epsilon)$ remains Littlestone based only on restricting $\epsilon$ as a function of Littlestone dimension $d$.

11. Examples

In this section and the next we work out some simple and some less simple examples relevant to the paper. Any singleton is a good set, in any graph (and for any value of $\epsilon$).

11.1. Cliques

There is a slight asymmetry between cliques and anticliques since we assume that graphs have no loops. If $G$ is an anticlique, every finite subset is $\epsilon$-good. But also notice that every good subset of $G$ realizes a type which is already realized by a vertex in $G$. So $G_{\infty}(\epsilon)=G$.

If $G$ is a clique, every singleton and every sufficiently large finite set $X\subseteq G$, i.e. of size at least $\epsilon^{-1}$, is $\epsilon$-good, indeed $\epsilon$-excellent. There exist excellent sets which realize types not already realized in $G$, namely that of connecting to every $v\in G$ (since no vertex of $G$ is connected to itself). Thus, if $\epsilon<1/2$ and $|G|>1/\epsilon$, we will have that $G_{1}(\epsilon)$ is $G$ along with one new vertex connected to all elements of $G$ and to itself, and $G_{n}(\epsilon)=G_{n+1}(\epsilon)$ for all $n\geq 1$.

11.2. Matchings

Suppose $\mathcal{H}$ is a hypothesis class representing an infinite matching, i.e. $\mathcal{H}=\{h_{x}:x\in X\}$ and $h_{x}(z)=1$ iff $z=x$. If $H\subseteq\mathcal{H}$ is such that $|H|>\epsilon^{-1}$ then $H$ is $\epsilon$-good. Thus, any singleton and any sufficiently large finite subset of $\mathcal{H}$ is good. Note that, for sufficiently small $\epsilon$, all good sets which are not singletons represent the constant-zero function.

Similarly to §11.1, in an infinite matching, for any $\epsilon$, $\mathcal{H}_{1}$ is $\mathcal{H}$ adjoin one new hypothesis, the constant-zero function; and $\mathcal{H}_{n}=\mathcal{H}_{n+1}$ for all $n\geq 1$.

A similar analysis holds for finite matchings, where say $|X|=|\mathcal{H}|\in\mathbb{N}$. If $|\mathcal{H}|\leq 1/\epsilon$, there are no new good sets, and if $|\mathcal{H}|>1/\epsilon$, the constant-zero function appears. Note that in this second case, the constant-zero function is what we have called an $\epsilon$-type, but it is not finitely realizable in $\mathcal{H}$ since there is no function $h\in\mathcal{H}$ which agrees with it on the finite set $X$. This example continues in §12.4.

11.3. Random graphs

We say “$G$ is an infinite random graph” to mean that $G$ satisfies the model-theoretic extension axioms $T_{\operatorname{rg}}$.161616The axioms $T_{\operatorname{rg}}$ say: for each $n$, there is an axiom saying there are at least $n$ elements, and for each $n,m$ there is an axioms saying that for any two disjoint sets of $n$ and $m$ elements respectively, there is an element connected to all vertices in the first set and none in the second. There is a unique countable model up to isomorphism, which is isomorphic with probability 1 to the graph obtained by starting with countably many vertices and a fair coin and flipping the coin for each pair to determine whether or not this is an edge. This explains the model-theoretic name “random graph”; however, with the axiomatic approach, one has models of arbitrarily large infinite sizes.

Suppose $G$ is an infinite random graph. Then:

1. (1)

any singleton is a good set. 2. (2)

any finite $X\subseteq G$ of even size is not a good set. 3. (3)

any finite $X\subseteq G$ of odd size $2k+1$, $k\geq 1$ is not $\frac{k}{2k+1}$-good.

Thus, if $\epsilon\leq\frac{1}{3}$, the only finite sets which are $\epsilon$-good are the singletons. This follows by the extension axioms, and the fact that $\frac{k}{2k+1}$ attains its minimum value at $k=1$. Hence for $\epsilon\leq\frac{1}{3}$, $G=G_{\infty}(\epsilon)$.

11.4. Half-graphs

Suppose $G$ is an infinite half-graph, that is, $V$ is the disjoint union $A=\{a_{i}:i\in\mathbb{N}\}\cup B=\{b_{j}:j\in\mathbb{N}\}$, and $a_{i}\sim b_{j}$ iff $i<j$. We make no assumptions about other edges.171717By convention, “half-graph” we will mean that the bi-induced pattern is a half graph, and will make no assumptions about the edges among the $a$’s or among the $b$’s. Then:

1. (1)

any singleton is a good set. 2. (2)

if $X\subseteq G$ is finite but not a singleton, and $X\subseteq A$ or $X\subseteq B$, then $X$ is not $\frac{1}{3}$-good. 3. (3)

if $X\subseteq G$ is finite, and $X\cap A\neq\emptyset$ and $X\cap B\neq\emptyset$, then $X$ is not $\frac{1}{6}$-good.

Thus, if $\epsilon\leq\frac{1}{6}$, the only finite sets which are $\epsilon$-good are the singletons. Here also $G(\epsilon)=G_{\infty}(\epsilon)$.

Proof.

Since Littlestone is central to the paper and is characterized via half-graphs, we work this out precisely. For (1), we use the fact that all singletons are $\epsilon$-good.

For (2), if $X\subseteq A$, let $i_{m}$ be the median of $\{i:a_{i}\in X\cap A\}$. Then, $b_{i_{m}}$ partitions $X$ in a $\frac{1}{3}$-balanced way, since the smallest part has size

[TABLE]

Therefore, $X$ is not $\frac{1}{3}$-good. The case where $X\subseteq B$ is analogous.

For (3), one of $A\cap X$ and $B\cap X$ has size at least $\frac{1}{2}|X|$. Without loss of generality, let it be $A\cap X$. As in (2), there some $b\in B$ splitting $A\cap X$ in an $\frac{1}{3}$-balanced way. Then, $b$ partitions $X$ in a $\frac{1}{6}$-balanced way, since the minimum of

[TABLE]

has size at least $\frac{|A\cap X|}{3}\geq\frac{|X|}{6}$. ∎

In the next examples, it will be convenient to display types as $(A,B)$ where $A\cup B=X$, $A$ are the elements labeled $1$ and $B$ are the elements labeled [math].

11.5. An example where $\mathcal{H}_{0}\neq\mathcal{H}_{1}\neq\mathcal{H}_{2}$.

We work out a basic example to show that saturation can take more than one step. This will be generalized in the next example, but may be simpler to read in this first case.

Let $\epsilon\in(\frac{1}{3},\frac{1}{2})$, and consider the following hypothesis class $\mathcal{H}$:

• •

$\mathcal{X}=\{x_{1},x_{2},x_{3},x_{4}\}$

• •

$\mathcal{H}=\{h_{(i,j)}:1\leq i<j\leq 4\}$

• •

$h_{(i,j)}(a_{\ell})=1$ iff $\ell\in\{i,j\}$.

In words, the three hypotheses with $i$ in the subscript connect to $x_{i}$, and to exactly one of the other $x_{j}$’s. From $\mathcal{H}_{0}$ to $\mathcal{H}_{1}$ to $\mathcal{H}_{2}$, the types realized over $\mathcal{X}$ progress from 2-element subsets, to 1-element subsets, to the empty subset.

• •

Analysis of $\mathcal{H}_{0}=\mathcal{H}$: types realized by $\mathcal{H}$ over $\mathcal{X}$ are $\{(A,B):|A|=2\}$.

• •

Analysis of $\mathcal{H}_{1}$: Additional to the previously realized types,

• –

The elements $H_{\ell}=\{h_{(i,j)}:\ell\in\{i,j\}\}$, for $1\leq\ell\leq 4$ were added, and each realizes some type $(A,B)$ with $|A|=1$. If $\ell=k$, then $e(x_{\ell},H_{\ell})=|H_{\ell}|$ and $H_{\ell}(x_{k})=1$. Otherwise, $e(x_{\ell},H_{\ell})=\frac{1}{3}|H_{\ell}|<\epsilon|H_{\ell}|$, and $H_{\ell}(a_{k})=0$. Therefore, $H_{\ell}$ realizes $(\{x_{\ell}\},\{x_{i}:i\neq\ell\})$.

• –

No element of $\mathcal{H}_{1}$ realizes $(\emptyset,X)$. Indeed, if some $\epsilon$-good set $S\subseteq\mathcal{H}_{0}$ of size $m$ realized this type, then $e(S,\mathcal{X})=2m$. By Pigeonhole principle, there is $x\in\mathcal{X}$ such that $e(x,S)\geq\frac{2m}{4}\geq\frac{1}{2}$, and $\mathbf{t}(x,S)=1$ must hold.

• •

Analysis of $\mathcal{H}_{2}$: $h=\{H_{1},H_{2},H_{3},H_{4}\}$ was added. Additional to the previously realized types, it realizes $(\emptyset,X)$ because $e(x_{i},h)=\frac{1}{4}|h|<\epsilon|h|$ for all $1\leq i\leq 4$.

To summarize this example: we started with a set $X$ of size $4$, our hypotheses corresponded to all $2$-element subsets of $X$; so notice that any $Y\subseteq X$ of size two is shattered. At stage zero, every $h\in\mathcal{H}$ has degree two. At stage one, we acquire some hypotheses of degree one, and at stage two, we acquire a hypothesis of degree zero. This descent of degree suggests why the $\operatorname{VC}$ and Littlestone dimension don’t increment even though we are adding new sets. This becomes quite a bit more general in the next example.

11.6. An example where $k$ extension steps are needed before saturation

Analogous to the previous example, we can construct finite classes $(X,\mathcal{H})$ that need an arbitrary finite number of steps to occur until saturation.

Lemma 11.1**.**

Let $k\in\mathbb{N}$, and $\epsilon\in(0,\frac{2}{2k-1})_{\mathbb{R}}$. Then, there exists $n\in\mathbb{N}$ and $(X,\mathcal{H}_{0})$ such that $|X|=n$ and $\mathcal{H}_{0}\subsetneq\mathcal{H}_{1}\subsetneq\ldots\subsetneq\mathcal{H}_{k}$.

Proof.

First, we can find some $n>2k-2$ such that $\epsilon\in(\frac{1}{n-k+1},\frac{2}{n})$, because

[TABLE]

Having chosen $n$, we can let $X=\{a_{i}:i\in[n]\}$ and $\mathcal{H}_{0}=\{b^{k}_{S}:\;S\in\binom{[n]}{k}\}$, where $b_{S}^{k}(a_{i})=[[i\in S]]$. The types realized by $\mathcal{H}_{0}$ over $X$ are $\{(A,B):|A|=k\}$. By the choice of $\epsilon$, we have $\mathcal{H}_{1}\supseteq\mathcal{H}_{0}\cup\{b_{S}^{k-1}:\;S\in\binom{[n]}{k-1}\}\text{ , where }b_{S}^{k-1}:=\{b_{S^{\prime}}^{k}:\;S\subset S^{\prime}\}$.

The elements $b_{S}^{k-1}$ are $\epsilon$-good sets of size $n-k+1$, realizing over $X$ the types $\{(A,B):|A|=k-1\}$, because $\epsilon>\frac{1}{n-k+1}$ allows for

[TABLE]

Hence, no $b_{S}^{k-1}$ is in $\mathcal{H}_{0}$, implying $\mathcal{H}_{0}\subsetneq\mathcal{H}_{1}$. The proof of $\mathcal{H}_{\ell-1}\subsetneq\mathcal{H}_{\ell}$ is similar for each $2\leq\ell\leq k$, and reveals which types are realized at each step:

• •

We can define $b_{S}^{k-\ell}:=\{b_{S^{\prime}}^{k-\ell+1}:S^{\prime}\in\binom{n}{k-\ell+1},\;\;S\subset S^{\prime}\}$ for each $S\in\binom{n}{k-\ell}$, which is an $\epsilon$-good set of $(n-k+\ell)$ elements in $\mathcal{H}_{\ell-1}$, by $\epsilon>\frac{1}{n-k+\ell}$ and

[TABLE]

• •

Hence $\mathcal{H}_{\ell}\supseteq\mathcal{H}_{\ell-1}\cup\{b_{S}^{k-\ell}:S\in\binom{n}{k-\ell}\}\supsetneq\mathcal{H}_{\ell-1}$, by the fact that elements in $\mathcal{H}_{\ell}\setminus\mathcal{H}_{\ell-1}$ realize $\{(A,B):|A|=k-\ell\}$, and by the claim below.

Claim 11.2**.**

For every $1\leq\ell\leq k$, the elements in $\mathcal{H}_{\ell-1}$ only realize types $\{(A,B):|A|\geq k-\ell+1\}$ over $X$.

Proof.

We use induction over $\ell$. For $\ell=1$, $\mathcal{H}_{0}$ indeed realizes types in $\{(A,B):|A|\geq k\}$, since $\mathcal{H}_{0}=\{b_{S}^{k}:S\in\binom{[n]}{k}\}$ and $e(b_{S}^{k},X)=k$ for all such $S$. For the inductive step, any element of $\mathcal{H}_{\ell-1}$ is of one of three kinds. For each, we argue the type realized is of size $\geq k-\ell+1$.

1. (1)

Elements $b_{S}^{k},\ldots,b_{S}^{k-\ell+1}$ have $e(b_{S}^{\cdot},X)\geq k,\ldots,k-\ell+1$, respectively. 2. (2)

Element $b_{S}^{p}\in\mathcal{H}_{p}$ added at step $p\leq\ell-2$, satisfies $e(b_{S}^{p},X)\geq k-\ell+2$, by inductive hypothesis. 3. (3)

An $\epsilon$-good set $S\subseteq\mathcal{H}_{\ell-2}$ added as $b\in\mathcal{H}_{\ell-1}$. If $|S|=m$, then $e(X,S)\geq(k-\ell+2)m$, since types realized by $\mathcal{H}_{\ell-2}$ are of size $\geq k-\ell+2$. There must exist $a_{i_{r}}\in X$ for $r\in[k-\ell+1]$, such that $e(a_{i_{r}},S)\geq\frac{2m}{n-k+\ell}>\frac{2}{n}$ for each $r$. Else, from counting in two ways, we get a contradiction

[TABLE]

Then, $\epsilon<2/n$ implies $\mathbf{t}(a_{i_{r}},b)=\mathbf{t}(a_{i_{r}},S)=1$ for all $r\in[k-\ell+1]$.

∎

This completes our proof that $\mathcal{H}_{0}\subsetneq\ldots\subsetneq\mathcal{H}_{k}$ holds for the constructed $(X,\mathcal{H}_{0})$. ∎

These examples give a family of Littlestone classes where the saturation construction does not stabilize until level $k$. Note that this is arranged at the cost of higher and higher initial, though obviously still finite, $\operatorname{VC}$-dimension.

Also note that $\operatorname{VC}(\mathcal{H}_{i})=\operatorname{Ldim}(\mathcal{H}_{i})=k$ for all $i<\omega$, and in particular $\operatorname{VC}(\mathcal{H}_{\infty}(\epsilon))=\operatorname{Ldim}(\mathcal{H}_{\infty}(\epsilon))=k$. On one hand, $\operatorname{VC}(\mathcal{H}_{i})\leq\operatorname{Ldim}(\mathcal{H}_{i})\leq k$ can be shown by noticing that all virtual elements realize types of size at most $k$. On the other hand, $\operatorname{Ldim}(\mathcal{H}_{i})\geq\operatorname{VC}(\mathcal{H}_{i})\geq\operatorname{VC}(\mathcal{H}_{0})\geq k$, because any subset

of $X$ of size $k$ can be shattered by elements $b_{S}^{k}\in\mathcal{H}_{0}$ as long as $|X|\geq 2k$ (a property of the above constructions, as long as $\epsilon<1/k$).

12. Examples for large $\epsilon$

In this section we prove by example that for “large” epsilon, the saturation of a Littlestone class can even have infinite VC dimension.

12.1. An example where VC-dimension goes to infinity

The following claim shows that if $\epsilon>0$ is large enough, it is possible for VC-dimension to increment at every step of saturation:

Example 12.1**.**

Consider the class $(X,\mathcal{H})$ where $X:=\mathbb{N}$, $\mathcal{H}_{0}:=\{b_{S}\subseteq\mathbb{N}:|S|=d\}$, and $b_{S}(i):=[[i\in S]]$ for any $S\subset\mathbb{N}$. Then, $\operatorname{VC}(\mathcal{H}_{0})=\operatorname{Ldim}(\mathcal{H}_{0})=d$. What’s more, if $\epsilon>\frac{1}{d+1}$, then $\operatorname{Ldim}(\mathcal{H}_{n})\geq\operatorname{VC}(\mathcal{H}_{n})\geq n+d$ for all $n<\omega$. In particular, $\operatorname{VC}(\mathcal{H}_{\infty}(\epsilon))=\omega$.

Proof.

To show $\operatorname{VC}(\mathcal{H}_{0})=\operatorname{Ldim}(\mathcal{H}_{0})=d$, all elements of $\mathcal{H}$ have degree $d$ and infinite codegree, so any set of $d$ elements in $X$ can be shattered. We show by induction on $n<\omega$ that $\mathcal{H}_{n}\supseteq\{b_{S}\subseteq\mathbb{N}:d\leq|S|\leq n+d\}$, which would imply $\operatorname{Ldim}(\mathcal{H}_{n})\geq\operatorname{VC}(\mathcal{H}_{n})\geq n+d$. The case $n=0$ holds by construction. Suppose $\mathcal{H}_{n}\supseteq\{b_{S}\subseteq\mathbb{N}:d\leq|S|\leq n+d\}$ for some $n<\omega$. Then, for all $A\subseteq\mathbb{N}$ with $|A|=n+d+1$, the set

[TABLE]

is of size $(n+d+1)$, and for all $i\in\mathbb{N}$, either $e(i,B_{A})=0$ when $i\not\in A$, or $e(i,B_{A})=n+d$ when $i\in A$. Since $\epsilon>\frac{1}{d+1}\geq\frac{1}{n+d}$, then $\mathbf{t}(i,B_{A})=[[i\in A]]$, is defined, $B_{A}$ is $\epsilon$-good, and $b_{A}\in\mathcal{H}_{n+1}$ is added. Thus, $\mathcal{H}_{n+1}\supseteq\{b_{S}:d\leq|S|\leq(n+d+1)\}$. To show that $\operatorname{Ldim}(\mathcal{H}_{n})\geq\operatorname{VC}(\mathcal{H}_{n})\geq n+d$, we use the same reasoning as for $\mathcal{H}_{0}$. ∎

To summarize a key idea of this example: Suppose $X$ is infinite and $\mathcal{H}$ is the set of (characteristic functions of) $k$-element subsets of $X$. Consider $\{a_{1},\dots,a_{n+1}\}\subseteq X$. We would like to realize some pattern of zeros and ones over it, call this pattern $\sigma:\{a_{1},\dots,a_{n+1}\}\rightarrow\{0,1\}$. By assumption on $\mathcal{H}$, for every $1\leq i\leq n+1$, there is a function $h_{i}$ which agrees with $\sigma$ on $\{x_{1},\dots,x_{n+1}\}\setminus\{x_{i}\}$. Consider $H=\{h_{i}:1\leq i\leq n+1\}$. (We can choose the $h_{i}$’s to be distinct, or we can make this a weighted set.) For every $i$, at least $n/(n+1)$ of the functions vote correctly. So if $\epsilon>1/n$, the majority vote of $H$ is well defined and agrees with $\sigma$. In this way we progressively extend the shattering to larger sets.

12.2. An example of a good set which is not an excellent set

Excellent sets play an important role in the sections on graphs. For more discussion and examples on this see [17]. Let $\epsilon=\frac{21}{60}$, and $G=(V,E)$ be the graph shown below, with $V=A\cup B$, $A=\{a_{i}\}_{i\in[6]}$, and $B=\{b_{j}\}_{j\in[6]}$.

While $A$ and $B$ are both $\varepsilon$-good, neither is $\varepsilon$-excellent. Indeed, all vertices have 0, 1, or 5 neighbors in $A$, and 0, 2 or 4 neighbors in $B$, and $\{\frac{i}{6}:i\in\{0,\ldots,6\}\setminus\{3\}\}\subseteq[0,\varepsilon)\cup(1-\varepsilon,1]$. Moreover, $\mathbf{t}(A,B)$ is not defined because $B$ is $\varepsilon$-good and $|\{a\in A:\mathbf{t}(a,B)=0\}|=|\{a\in A:\mathbf{t}(a,B)=1\}|=\frac{1}{2}\cdot|A|$. Similarly, $\mathbf{t}(B,A)$ is not defined because $A$ is $\varepsilon$-good and $|\{b\in B:\mathbf{t}(b,A)=0\}|=|\{b\in B:\mathbf{t}(b,A)=1\}|=\frac{1}{2}\cdot|B|$.

12.3. Extensions

Observe that in both cliques 11.1 and matchings 11.2, given any two good sets, there is a larger good set containing both of them. This is a property that all bounded degree graphs enjoy:

Proposition 12.2**.**

Let $G=(V,E)$ be an arbitrary infinite graph such that every vertex has degree at most $d\in\mathbb{N}$. Then, every finite $X\subseteq V$ with $|X|>\frac{d}{\epsilon}$ is $\epsilon$-good. Moreover, for any two $\epsilon$-good sets $X_{1}$ and $X_{2}$, there exists an $\epsilon$-good set $X_{3}$ such that $X_{1}\cup X_{2}\subseteq X_{3}$.

Proof.

For the first part, consider such vertex set $X$, and any $v\in V$. Since $\deg(v)\leq d$, $v$ partitions $X$ in an $\epsilon$-unbalanced way, since the smaller size of this partition is less than or equal than $d<\epsilon\cdot|X|$.

For the second part, consider two $\epsilon$-good sets $X_{1}$ and $X_{2}$. Infiniteness of $|V|$ implies there is a set $X_{3}\subset V$ containing $X_{1}\cup X_{2}$, possibly with some vertices added in order to have $|X_{3}|>\frac{d}{\epsilon}$ hold. By the first claim, $X_{3}$ is $\epsilon$-good. ∎

12.4. Realizability and the value of $\epsilon$

Suppose $\mathcal{H}$ is a hypothesis class and $r<1/\epsilon$. Observe that any function which is $r$-realized in $\mathcal{H}_{1}(\epsilon)$ is already realized in $\mathcal{H}$. [Why? By a union bound, see 7.2 for a more general result.] On the other hand, consider the class of singletons over $X=\{1,2,3,4,5\}$, $\epsilon=1/4$, and $r=5>1/\epsilon$. In this example, $H=\mathcal{H}$ (with the counting measure) is an $\epsilon$-good set, and $\mathbf{t}_{H}$ is the constant zero function, which is realized in $\mathcal{H}_{1}(\epsilon)$ and even $4$-realized in $\mathcal{H}$, but not realized, indeed not even $5$-realized, in $\mathcal{H}$.

13. Open questions

There are some very interesting combinatorial questions arising from this analysis around the regimes of $\epsilon$, which don’t have obvious analogues in infinitary notinos of saturation. Theorem 7.1 draws our attention to the case $1/Cd\geq\epsilon\geq 1/(d+1)$. The first few questions are about different aspects of this gap. In each case we assume $\operatorname{Ldim}(\mathcal{H})=\ell<\infty$ and hence $\operatorname{VC}(\mathcal{H})=d<\infty$, and in each case $\mathcal{H}_{\infty}$ means $\mathcal{H}_{\infty}(\epsilon)$. The constant $C$ is from 7.1.

Question 13.1**.**

If $\operatorname{Ldim}(\mathcal{H})<\infty$, can it happen that $\operatorname{Ldim}(\mathcal{H}_{\infty})=\infty$ but $\mathcal{H}_{\infty}$ still has the property that $\operatorname{VC}(\mathcal{H}_{\infty})<\infty$?

Question 13.2**.**

In general, what happens to $\operatorname{Ldim}(\mathcal{H}_{\infty}(\epsilon))$ when $1/Cd\leq\epsilon$?

It follows from the definitions that if ever $\mathcal{H}_{n}=\mathcal{H}_{n+1}$ then $\mathcal{H}_{n+1}=\mathcal{H}_{m}$ for all $m>n$. However, our only example above where $\mathcal{H}_{n}\neq\mathcal{H}_{n+1}$ for all $n<\omega$ is in the case where $\operatorname{VC}(\mathcal{H}_{\infty})=\infty$. The next question asks if this can happen in a tamer setting. (By the above, sufficiently small values of $\epsilon$ can be ruled out.)

We can also ask the parallel question for graphs $G$.

Question 13.3**.**

What happens to the Littlestone dimension of $\mathcal{G}_{\epsilon}(\infty)$ when $\mathcal{G}$ is a Littlestone class and $\epsilon$ is of intermediate range bounded as a function of the $\operatorname{VC}$ dimension $d$, per Remark 10.10?

Question 13.4**.**

Are there any other preservation theorems for the saturation with respect to other combinatorial complexity measures?