Optimal Prediction Using Expert Advice and Randomized Littlestone Dimension
Yuval Filmus, Steve Hanneke, Idan Mehalel, Shay Moran
TL;DR
This paper establishes the optimal mistake bounds for randomized online learning using Littlestone dimension, extending classical results and resolving longstanding open problems in prediction with expert advice.
Contribution
It introduces the randomized Littlestone dimension, characterizes optimal mistake bounds for randomized learners, and applies this to improve bounds in prediction with expert advice.
Findings
Optimal randomized mistake bound equals the randomized Littlestone dimension.
In the agnostic case, mistake bounds are characterized by $k + ilde{O}( oot{kd}) + d$.
Provides an optimal learning rule for prediction with expert advice, halving the deterministic mistake bound.
Abstract
A classical result in online learning characterizes the optimal mistake bound achievable by deterministic learners using the Littlestone dimension (Littlestone '88). We prove an analogous result for randomized learners: we show that the optimal expected mistake bound in learning a class equals its randomized Littlestone dimension, which is the largest for which there exists a tree shattered by whose average depth is . We further study optimal mistake bounds in the agnostic case, as a function of the number of mistakes made by the best function in , denoted by . We show that the optimal randomized mistake bound for learning a class with Littlestone dimension is . This also implies an optimal deterministic mistake bound of , thus resolving an open question which was…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMachine Learning and Algorithms · Imbalanced Data Classification Techniques · Advanced Bandit Algorithms Research
\forestset
default preamble=for tree=s sep+=1cm
Optimal Prediction Using Expert Advice
and
Randomized Littlestone Dimension
Yuval Filmus
The Henry and Marilyn Taub Faculty of Computer Science, Technion, Israel
Faculty of Mathematics, Technion, Israel
Steve Hanneke
Department of Computer Science, Purdue University, USA
Idan Mehalel
The Henry and Marilyn Taub Faculty of Computer Science, Technion, Israel
Shay Moran
Faculty of Mathematics, Technion, Israel
The Henry and Marilyn Taub Faculty of Computer Science, Technion, Israel
Google Research, Israel
Abstract
A classical result in online learning characterizes the optimal mistake bound achievable by deterministic learners using the Littlestone dimension (Littlestone ’88). We prove an analogous result for randomized learners: we show that the optimal expected mistake bound in learning a class equals its randomized Littlestone dimension, which we define as follows: it is the largest for which there exists a tree shattered by whose average depth is . We further study optimal mistake bounds in the agnostic case, as a function of the number of mistakes made by the best function in , denoted by . Towards this end we introduce the -Littlestone dimension and its randomized variant, and use them to characterize the optimal deterministic and randomized mistake bounds. Quantitatively, we show that the optimal randomized mistake bound for learning a class with Littlestone dimension is (equivalently, the optimal regret is ). This also implies an optimal deterministic mistake bound of , thus resolving an open question which was studied by Auer and Long [’99].
As an application of our theory, we revisit the classical problem of prediction using expert advice: about 30 years ago Cesa-Bianchi, Freund, Haussler, Helmbold, Schapire and Warmuth studied prediction using expert advice, provided that the best among the experts makes at most mistakes, and asked what are the optimal mistake bounds (as a function of and ). Cesa-Bianchi, Freund, Helmbold, and Warmuth [’93, ’96] provided a nearly optimal bound for deterministic learners, and left the randomized case as an open problem. We resolve this question by providing an optimal learning rule in the randomized case, and showing that its expected mistake bound equals half of the deterministic bound of Cesa-Bianchi et al. [’93, ’96], up to negligible additive terms. In contrast with previous works by Abernethy, Langford, and Warmuth [’06], and by Brânzei and Peres [’19], our result applies to all pairs , and does so via a unified analysis using the randomized Littlestone dimension.
In our proofs we develop and use optimal learning rules, which can be seen as natural variants of the Standard Optimal Algorithm () of Littlestone: a weighted variant in the agnostic case, and a probabilistic variant in the randomized case. We conclude the paper with suggested directions for future research and open questions.
Contents
1 Introduction
A recurring phenomenon in learning theory is that different notions of learnability are captured by combinatorial parameters. Notable examples include the Vapnik–Chervonenkis (VC) dimension which characterizes PAC learnability [VC74, BEHW89] and the Littlestone dimension which characterizes online learnability [Lit88, BDPSS09]. Other examples include the Daniely–Shalev-Shwartz and Natarajan dimensions in multiclass PAC learning [Nat89, DSS14, BCD*+*22], the star number, disagreement coefficient, and inference dimension in interactive learning [Han14, HY15, KLMZ17], the statistical query dimension in learning with statistical queries [Fel17], the representation dimension, one-way communication complexity, and Littlestone dimension in differentially private learning [FX15, BNS19, ABL*+*22], and others.
One of the simplest and most appealing characterizations is that of online learnability by the Littlestone dimension. In his seminal work, Nick Littlestone proved that the optimal mistake-bound in online learning a class is exactly the Littlestone dimension of [Lit88]. Thus, not only does the Littlestone dimension qualitatively captures online learnability, it also provides an exact quantitative characterization of the best possible mistake bound. This distinguishes the Littlestone dimension from other dimensions in learning theory, which typically only provide asymptotic bounds on the learning complexity.
However, the exact quantitative characterization of the optimal mistake bound by the Littlestone dimension applies only in the noiseless realizable setting and only for deterministic learners. In particular, it does not apply in the more general and well-studied setting of agnostic online learning. The reason it does not apply is twofold: (i) because the agnostic setting allows for non-realizable sequences, and (ii) because randomized learners are in fact necessary.111Randomized learners are necessary in the following sense: any agnostic online learner for a class must be randomized, provided that contains at least two functions [Cov65], see also [SSBD14, Chapter 21.2]. This suggests the following question, which guides this work:
Is there a natural dimension which captures the optimal expected mistake bound in learning a class using randomized learners? How about the agnostic setting when there is no which is consistent with input data?
The main contribution of this work formalizes and proves affirmative answers to these questions. Some of the technical material is omitted from this manuscript and can be found in the full version which is accessible online in [FHMM22].
Organization.
In the next section we present the main results of this work. Then, in Section 3 we provide a short technical overview, where we outline the main ideas we use in our proofs. The remaining sections contain the complete proofs.
2 Main results
This section assumes familiarity with standard definitions and terminology from online learning. We refer the unfamiliar reader to Section 4, which introduces the online learning model and related basic definitions in a self-contained manner.
2.1 Realizable Case
In his seminal work from 1988, Nick Littlestone studied the optimal mistake bound in online learning an hypothesis class by deterministic learning rules in the realizable setting [Lit88]; that is, under the assumption that the input data sequence is consistent with a function .
Littlestone dimension.
Let be the domain, and let be a class of “” predictors. The Littlestone dimension of , denoted , is the maximal depth of a binary complete decision tree which is shattered by . That is, a decision tree whose nodes are associated with points from and whose edges are associated with labels from such that each of the branches (root-to-leaf paths) in is realized by some .
Littlestone proved that the optimal mistake bound achievable by deterministic learners equals the Littlestone dimension:
Theorem 2.1** (Deterministic Mistake Bound [Lit88]).**
The optimal deterministic mistake bound in online learning in the realizable setting is equal to its Littlestone dimension, .
Littlestone further described a natural deterministic learning rule, which he dubbed the Standard Optimal Algorithm (), that makes at most mistakes on every realizable input sequence.
Randomized Littlestone dimension.
Our first main result shows that a natural probabilistic variant of the Littlestone dimension characterizes the optimal expected mistake bound for randomized learners.
Definition 2.2** (Randomized Littlestone Dimension).**
Let be binary tree, and consider a random walk on that starts at the root, goes to the left or right child with probability , and continues recursively in the same manner until reaching a leaf. Let denote the expected length of a random branch which is produced by this process.
The randomized Littlestone dimension of a class , denoted by , is defined by
To compare the randomized Littlestone dimension with the Littlestone dimension, notice that the Littlestone dimension is equal to , where is the minimum length of a branch in . Thus, the difference is that in we take the expected depth rather than the minimal depth, and multiply by a factor of .222From a learning theoretic perspective it is easy to see that , because randomized learners are more general than deterministic ones. Interestingly, this inequality is less obvious from a combinatorial perspective: indeed, for every fixed tree we have that (because the expected length of a branch is at least the minimal length), but it is not a priori clear why the inequality is reversed when is replaced by and we take supremum over all shattered trees.
Theorem 2.3** (Main Result (i): Randomized Mistake Bound).**
The optimal randomized mistake bound in online learning in the realizable setting is equal to its randomized Littlestone dimension, .
We also provide an optimal randomized learning rule which can be seen as a probabilistic adaptation of Littlestone’s classical algorithm. See Section 3.1 for a brief overview, and Section 5.1 for the proof.
The connection between online learning problems and random walks was identified in the online learning literature [AWY08, LS14a, GPS16]. [AWY08] asked, conceptually, how general is this connection. Our results show that it is indeed quite general, in the sense that it yields the optimal algorithm for every hypothesis class.
2.2 Agnostic Case
We next consider the agnostic setting, in which we no longer assume that the input sequence of examples is consistent with . Our second main result characterizes the optimal expected mistake bound in this setting.
A common approach for handling the agnostic case is to assume a bounded horizon and analyze the regret. That is, it is assumed that the length of the input sequence (called the horizon) is a given parameter , and the goal is to design learning rules whose mistake bound is competitive with that of the best up to an additive term which is negligible in (this term is called the regret of the algorithm).
The bounded horizon assumption simplifies the design of learning rules, by allowing them to depend on . A notable example is the celebrated Multiplicative Weights (MW) learning rule, whose learning rate depends on . This assumption can then be lifted by standard doubling tricks.333E.g. start by running the algorithm with , and double when reaching the ’st example.
The -realizable setting.
In this work we consider an alternative approach: instead of assuming a bound on the horizon, we assume a bound on the number of mistakes made by the best function in the class. Notice that this assumption can also be lifted by suitable doubling tricks as we demonstrate in Section 2.4, where we also extend our results to the bounded-horizon setting.
The upshot of this approach is that it allows for a precise combinatorial characterization of the optimal mistake bound via a natural generalization of the Littlestone dimension.
2.2.1 -Littlestone Dimension
Let be an hypothesis class, and let . A sequence of examples is -realizable by if there exists such that for at most indices . In the -realizable setting we assume that the input sequence given to the learner is -realizable. Notice that the case amounts to realizability by . We say that a decision tree is -shattered by if every branch is -realizable by . The corresponding deterministic and randomized -Littlestone dimensions of a class are
[TABLE]
Theorem 2.4** (Main Result (ii): -Littlestone Dimension).**
Let be an hypothesis class.
The optimal deterministic mistake bound in online learning in the -realizable setting equals its -Littlestone dimension, .
The optimal randomized mistake bound in online learning in the -realizable setting equals its -randomized Littlestone dimension, .
We also provide optimal learning rules which can be seen as weighted variants of Littlestone’s classical algorithm. See Section 3.1 for a brief overview and Section 8 for the proof.
As a consequence of this perspective, we prove the following theorem which provides tight regret bounds in terms of the Littlestone dimension.
Theorem 2.5** (Main Result (iii): Optimal Regret Bounds for Littlestone Classes).**
Let be an hypothesis class and let . Then
In particular, the optimal regret in online learning is , where is the number of mistakes made by the best function in .
This improves and refines over results by [AL99, ABED*+*21]. The work by [ABED*+*21] determined an optimal regret bound of , where is the time horizon. The above bound refines it by replacing with . The work by [AL99] studies the optimal deterministic mistake bound in online learning in the -realizable setting. Theorem 2.5 and the deterministic lower bound of [LW94] imply that the deterministic mistake bound is
[TABLE]
This improves over [AL99, Theorem 4.4], which gives an upper bound of for every . See Section 8.4 for the proof of Theorem 2.5.
2.3 Prediction Using Expert Advice
In this section, we consider the problem of prediction using expert advice [Vov90, LW94]. This problem studies a repeated guessing game between a learner and an adversary. In each round of the game, the learner needs to guess the label that the adversary chooses. In order to do so, the learner can use the advice of experts. Formally, each round in the game proceeds as follows:
- (i)
The experts present predictions . 2. (ii)
The learner predicts a value . 3. (iii)
The adversary reveals the true answer , and the learner suffers the loss .
The value should be understood as the probability (over the learner’s randomness) of predicting . Notice that the adversary only gets to see , which reflects the assumption that the adversary does not know the learner’s internal randomness. Notice also that the suffered loss exactly captures the probability that the learner makes a mistake. The above is a standard way to model randomized learners in online learning, see e.g. [Sha12, Haz19, CBL06]. If for all , then the learner is deterministic, in which case is the binary indicator for whether the learner made a mistake.
We focus here on the -realizable setting, which was suggested by [CBFHW96, CBFH*+*97] and further studied by [ALW06, MS10, BP19]. Here, the adversary must choose the answers so that at least one of the experts makes at most mistakes. That is, there must exist an expert such that for at most many indices .
The goal is to determine the optimal loss of the learner as a function of and . Let denote the optimal loss of a deterministic learner and denote the optimal loss of a (possibly) randomized learner.444Note that we assume here that is known to the learner and that the horizon (i.e. number of rounds in the game) might be unbounded. In Section 2.4.2 below we explain how to extend our results to the complementing cases.
The starting point is the basic fact555One might be tempted to interpret these inequalities as implying that and are nearly the same. However, the multiplicative gap of can be significant. For example, a randomized learner with a non-trivial error rate of corresponds to a deterministic learner with error-rate. The latter is trivially achieved by a random guess. For the same reason, sublinear regret guarantees can only be achieved by randomized learners, although they are “just” a factor of better than deterministic learners, see e.g. [CBL06, Sha12, Haz19]. that
[TABLE]
In their seminal work, Cesa-Bianchi et al. [CBFH*+*97] exhibited a randomized algorithm which witnesses that in the regime when or , the lower bound in Equation 1 is tight up to a relative factor of , (See their Theorem 4.4.3).
In a follow up work, [CBFHW96] aimed to find optimal deterministic and randomized algorithms. They found a nearly optimal deterministic algorithm called binomial weights, which is optimal (up to an additive constant) when is small enough. The main problem they left open is whether there is a randomized learner with loss exactly half the loss of their binomial weights algorithm (plus, maybe, a constant). Below we show that the answer to this question is negative, and find tight guarantees on the second-order term in , in terms of the performances of their algorithm.
Nearly 10 years later, Abernathy, Langford and Warmuth [ALW06] showed that for every and every , where is a universal constant (independent of ), thus showing that in the regime when the additive negligible term is indeed a universal constant (independent of ).
More recently, Brânzei and Peres [BP19] showed that for , while quantitatively improving upon the bounds given by [CBFH*+*97] in this regime.
In the next theorem we provide guarantees on for all and , which are tight when , thus fully resolving the question raised by [CBFHW96].666When , . Our lower bound shows that the second-order term tends to infinity when . The latter shows that the result by [ALW06] does not apply for general .
Theorem 2.6** (Main Result (iv): Bounds for Randomized Predictors).**
Let denote the optimal expected mistake bound for prediction using expert advice in the -realizable setting when there are experts, and let denote the mistake bound of the binomial weights algorithm. For all and ,
Furthermore, the error term cannot be improved for :
We prove the upper bound in Section 9.2, and the lower bound in Section 9.3. Both bounds are proved using the randomized -Littlestone dimension. A special case of Theorem 2.5 states that , where the upper bound in this quantitative bound was first proved in [CBFH*+*97].
Using Theorem 2.6 and the bounds of [CBFHW96], we obtain the following corollary.
Corollary 2.7**.**
For all and ,
[TABLE]
This also follows from the results of [Vov90] for randomized learners.777In the COLT 2023 proceedings version of this paper, this corollary was unintentionally presented as brand new.
Additional Related Work.
Different variants of the experts problem have been extensively studied in the past 30 years and various techniques for bounding the optimal regret and mistake bounds were developed throughout the years, such as sequential Rademacher complexity [RSS12, RS14], drifting games [MS10, LS14b], and the Hedge setting [AWY08, FS97]. However, those techniques are seemingly tailored for randomized proper learners (i.e., learners that predict using a distribution over the experts which is updated at the end of each round), and proper learners are inherently suboptimal for the experts problem, even in the realizable case, as proven in the full version of this paper [FHMM22]. [AWY08] identified the optimal proper algorithm, using a random walk analysis, which is similar to our characterization results. It will be interesting to investigate whether variations of these techniques can reproduce or even improve the bounds in this work.
2.4 Variations
2.4.1 Bounded Horizon
Consider learning in the -realizable setting, and let denote the optimal expected mistake bound. In particular, this means that the adversary can force mistakes in expectation on any randomized learner. This would be tolerable if in order to do so the adversary must use many examples, say . Indeed, this would mean that the learner makes only one mistake per a thousand examples (amortized), which is rather good.
This raises the question to what extent does capture the optimal mistake bound under the additional assumption that the horizon is bounded by a given . A bounded horizon is often assumed in the online learning literature, and in fact this question was explicitly asked by [CBFH*+*97] in the special case of prediction using expert advice.
Let denote the optimal expected mistake bound in the -realizable setting with horizon bounded by . The following result shows that provides an excellent approximation of ; in particular, the scenario described above is impossible.
Theorem 2.8** (Main Result (v): Bounded vs Unbounded Horizon).**
Let be an hypothesis class. Let denote the optimal expected mistake bound in online learning in the -realizable setting, and let denote the optimal expected mistake bound under the additional assumption that the input sequence has length at most . Then,
Long horizon.* If then*
Short Horizon.* If then*
and if then .
The upper bounds in Theorem 2.8 follow from basic facts: indeed, holds because assuming a bounded horizon restricts the adversary, and follows by guessing each label uniformly at random. The lower bounds are more challenging, and our proofs of them relies heavily on the randomized Littlestone dimension.
Our proof of Theorem 2.8 appears in Section 7.3. The proof relies on a simple extension of our characterization to this setting: consider the following modification of the Littlestone dimension and its randomized variant:
[TABLE]
The bounded randomized Littlestone dimension gives the precise mistake bound in this setting:
Theorem 2.9** (Optimal Mistake Bounds: Bounded Horizon).**
Let be an hypothesis class.
The optimal deterministic mistake bound in online learning in the -realizable setting with horizon equals its bounded -Littlestone dimension, .888Trivially, . 2. 2.
The optimal randomized mistake bound in online learning in the -realizable setting with horizon equals its bounded -randomized Littlestone dimension, .
We prove Theorem 2.9 in Section 7.1.
Prediction using Expert Advice.
Also the problem of prediction using expert advice is often considered when the number of rounds is bounded (e.g. [CBFH*+*97]). Let be the optimal loss of the learner when the number of rounds is . By a simple reduction to Theorem 2.9 we show that
[TABLE]
The exact bounds are as in Theorem 2.9 when replacing and with and .
2.4.2 Adaptive Algorithms
The analysis in much of this work considers the case where the learning algorithm may depend explicitly on a bound on the number of mistakes of the best hypothesis (or expert). However, it is also desirable to study mistake bounds achievable adaptively: that is, by a single algorithm that applies to all . We present here one simple approach to obtaining such an algorithm, with a corresponding mistake bound. However, the bound we obtain may likely be improvable, and generally we leave the question of obtaining a tightest possible adaptively-achievable mistake bound as an open problem.
Theorem 2.10**.**
There is an adaptive algorithm (i.e., which has no knowledge of ) such that, for every -realizable sequence for , its expected number of mistakes is at most
[TABLE]
In the special case of the general experts setting, since we know that , we obtain the following bound on the expected number of mistakes:
[TABLE]
In particular, combining this with Theorem 2.6, we find that this algorithm adaptively still achieves an expected number of mistakes .
On the other hand, in the case of concept classes with a bounded Littlestone dimension , we know from Theorem 2.5 that
[TABLE]
Theorem 2.10 implies that the adaptive procedure nearly preserves the form of this upper bound, guaranteeing a slightly larger bound of the form
[TABLE]
Our proof of Theorem 2.10 appears in Section 8.5. The adaptive technique we propose involves using an experts algorithm of [KvE15] named , with experts defined by the optimal randomized algorithm for the -realizable setting, for all values of .
3 Technical Overview
In its greatest generality, online prediction is a game involving two randomized parties, an adversary who is producing examples, and a learner who is trying to correctly predict the labels of all or most of these examples. In the realizable case, the adversary is moreover constrained by an hypothesis class which must be adhered to.
Various techniques are used in the literature to analyze this sophisticated setting. On the one hand, learning rules show which hypothesis classes lend themselves to learning, and on the other hand, strategies for the adversary put limitations on what can be learned, and at what cost.
In this work, we identify the combinatorial core behind many settings of online learning. In this, we follow up on Nick Littlestone’s classical work on deterministic online learning, as well as on other classical work in learning theory such as that the foundational work of Vapnik and Chervonenkis.
Reducing the messy probabilistic setting of online learning to the clean combinatorial setting of shattered trees enables us to tackle open questions about prediction using expert advice, which are hard to approach directly.
3.1 Combinatorial Characterizations
The Littlestone dimension of an hypothesis class is the maximal depth of a complete binary tree which is shattered by . A tree of depth easily translates into a strategy for the adversary which forces the learner to make mistakes. In other words, a tree shattered by is an obvious obstacle to learning .
The magic of Littlestone dimension is the opposite direction: Littlestone’s learning rule makes at most mistakes, showing that trees shattered by are the only obstacle for learning . This is a common phenomenon in mathematics: an obvious necessary condition turns out to be (less obviously) sufficient.
Defining the randomized Littlestone dimension.
In order to motivate the definition of the randomized Littlestone dimension, let us first examine the (deterministic) Littlestone dimension. Given a tree shattered by , the adversary executes the following strategy, starting at the root:
At an internal node labeled , ask the learner for the label of , and follow the opposite edge.
This strategy follows a branch of , and forces the learner to make a mistake in each round. The total number of mistakes which the adversary can guarantee is precisely , the minimum length of a branch in . The resulting input sequence is realizable by since is shattered by .
The definition of the randomized Littlestone dimension follows a similar approach, but uses a different strategy for the adversary:
At an internal node labeled , ask the learner for the label of , and follow a random edge.
This strategy also follows a branch of , and it forces the learner to make half a mistake in each round, in expectation.999Recall that we model a randomized learner as a learner which makes a “soft” prediction ; if the true label is , then the learner’s loss is . When we choose the label at random, the expected loss is regardless of . The total expected number of mistakes is , where is the expected length of a random branch of .
We define the randomized Littlestone dimension by considering all such adversary strategies:
[TABLE]
Extending the Standard Optimal Algorithm.
Littlestone’s Standard Optimal Algorithm () makes at most mistakes on any realizable input sequence. The algorithm is very simple. It maintains a subset of which consists of all hypotheses which are consistent with the data seen so far. Given a sample , one of the following must hold, where is the subset of consisting of all hypotheses assigning to the label :
. The learner predicts . 2. 2.
. The learner predicts .
One of these cases must hold, since otherwise we could construct a tree of depth shattered by . Each time that the learner makes a mistake, decreases by , and so the learner makes at most mistakes.
Our randomized extension of , which we call , follows a very similar strategy. It maintains in the same way. Given a sample , we want to make a prediction which “covers all bases”, that is, results in a good outcome for the learner whatever the correct label is. Given a prediction , the adversary can guarantee a loss of
[TABLE]
For the optimal choice of , this quantity is at most , as we show in Section 5.1.
The -realizable setting and weighted .
The -realizable setting is handled similarly. In the definition of randomized Littlestone dimension, instead of requiring the tree to be shattered, it suffices for it to be -shattered, since the adversary need only produce an input sequence which is -realizable.
The main novelty in this setting is a weighted analog of the learning rule. This weighted rule relates to the classical in a similar way like the Weighted Majority algorithm relates to Halving. In particular, it keeps track, for each hypothesis, how many more mistakes are allowed. Accordingly, we consider the more generalized setting of weighted hypothesis classes. These are hypothesis classes in which each hypothesis has a “mistake budget”. The definition of randomized Littlestone dimension extends to this setting, and allows us to generalize to the randomized agnostic setting.
3.2 Quasi-balanced Trees
Given an hypothesis class , how does an optimal strategy for the adversary look like? Such a strategy must make the analysis of tight, and in particular, if the first sample it asks is , then
[TABLE]
where is the prediction of the learner.101010Strictly optimal strategies do not always exist, and even when they do, they might require an unbounded number of rounds. For the sake of exposition we gloss over these difficulties.
The strategy of the adversary naturally corresponds to a tree which is shattered by : the root is labeled , and the edge labeled leads to a tree corresponding to an optimal strategy for . Suppose that we further assign weights to the edges touching the root: the [math]-edge gets the weight , and the -edge gets the weight . If we assign weights to the remaining edges recursively then the resulting tree satisfies the following property:
Every branch has the same total weight .
More generally, a tree is quasi-balanced if we can assign non-negative weight to its edges such that (i) the weights of the two edges emanating from a vertex sum to , and (ii) all branches have the same total weight (which must be ). If a tree is quasi-balanced then the weight assignment turns out to be unique.
A tree in which all branches have the same depth is quasi-balanced, but the class of quasi-balanced trees is a lot richer, including for example the path appearing in Figure 1.
There is a simple criterion for quasi-balancedness:
A tree is quasi-balanced if and only if it is monotone: if is a descendant of then , where is the subtree rooted at .
Since the loss guaranteed by an adversary following the strategy corresponding to a tree is , it is clear that the best strategy is always monotone. This argument shows that
[TABLE]
In other words, it suffices to consider only quasi-balanced trees when defining the randomized Littlestone dimension. This is the randomized counterpart of a trivial property of the Littlestone dimension: in order to define the Littlestone dimension, it suffices to consider balanced trees, that is, trees in which all branches have the same length. We can view quasi-balancedness as a relaxation of strict balancedness.
Concentration of expected branch length.
The randomized Littlestone dimension is defined in terms of the expected branch length. However, several of our results require knowledge of the distribution of the branch length.
For example, Theorem 2.8 states that rounds are needed in order for the adversary to guarantee a loss of . The number of rounds corresponds to the depth of the tree, and so the natural way to prove such a result would be to start with a tree satisfying , and prune it to depth . We would like to say that this does not reduce the expected branch length by much, since the length of most branches does not exceed by much. Other applications such as prediction using expert advice need concentration from the other side (the length of most branches does not fall behind by much).
It is possible to construct trees for which the length of a random branch isn’t concentrated around its expectation. For example, we can take an infinite path which, every so often, splits into a deep complete binary tree. If we are careful, we can guarantee that the expected branch length is finite but its variance is infinite.
At this point, quasi-balancedness comes to the rescue. The monotonicity property of quasi-balanced trees implies that the choice of an edge at every step of a random branch does not affect the final length by much. Consequently, Azuma’s inequality (a version of Chernoff’s inequality for martingales) shows that for quasi-balanced trees, the length of a random branch is strongly concentrated around its expectation. This simple observation drives several of our strongest results.
3.3 Prediction using Expert Advice
At first, the setting of prediction using expert advice looks similar, but not identical, to our setting. However, it turns out that it is actually a special case of our setting, for a specific hypothesis class known as the universal hypothesis class .
The class contains different hypotheses, which correspond to the experts. For each possible set of predictions there is a corresponding element in the domain. In other words, the domain is , and the hypotheses in are the projections .
With this equivalence in place, we can apply the theory we have developed so far to analyze prediction using expert advice. Our main result concerning this setting, Theorem 2.6, consists of an upper bound on , and a lower bound on .
We start with the upper bound on . In view of the equivalence above, we want to bound the expected branch length of any tree which is -shattered by . We can assume that is quasi-balanced, and so the length of a random branch of is roughly . If were strictly balanced, then a random branch would be -realizable by with probability at most
[TABLE]
[CBFHW96] have shown that the largest value of for which this quantity is at least , which we denote by , provides the state-of-the-art upper bound on . Since is only quasi-balanced, we get a slightly worse bound.
A nice proof of the lower bound on is given by identifying the optimal tree. Intuitively, it seems obvious that rounds in which both experts make the same prediction are “wasteful”, and we can show this formally. By symmetry, we can assume that the first expert always predicts [math] and that the second expert always predicts . We can construct the corresponding tree explicitly, and conclude that
[TABLE]
The proof of this result can be found in the full version [FHMM22] of this paper.
4 Background and Basic Definitions
Unless stated otherwise, our logarithms are base .
Online Learning.
Let be a set called the domain, and be a set called the label set. In this work we focus on binary classification, and thus . A pair is called an example, and an element is called an instance or an unlabeled example. A function is called a hypothesis or a concept. A hypothesis class, or a concept class, is a set . A sequence of examples is said to be realizable by if there exists such that for all .
Online learning [SSBD14, CBFH*+*97] is a repeated game between a learner and an adversary. Each round in the game proceeds as follows:
- (i)
The adversary sends the learner an unlabeled example . 2. (ii)
The learner predicts a value and reveals it to the adversary. 3. (iii)
The adversary reveals the true label , and the learner suffers the loss .
The value should be understood as the probability (over the learner’s randomness) of predicting . Notice that the adversary only gets to see , which reflects the assumption that the adversary does not know the learner’s internal randomness. Notice also that the suffered loss exactly captures the probability that the learner makes a mistake. The above is a standard way to model randomized learners in online learning, see e.g. [Sha12]. If for all , then the learner is deterministic, in which case is the binary indicator for whether the learner made a mistake.
We model learners as functions . Given a learning rule and an input sequence of examples , we denote the (expected) number of mistakes makes on by
[TABLE]
where is the prediction of the learner on the ’th example.
An hypothesis class is online learnable (or learnable) if there exists a finite bound and a learning rule such that for any input sequence which is realizable by it holds that . We define the optimal randomized mistake bound of to be
[TABLE]
where the infimum is taken over all learning rules, and the supremum is taken over all realizable input sequences .
We denote by the optimal deterministic mistake bound of . That is, is defined in the same way as , with the additional restriction that must be deterministic (that is, the output must be in ).
When , the set of realizable input sequences is empty, and therefore the supremum is not defined. It is technically convenient to deal with this special case by defining . When the context is clear, we may sometimes refer to the deterministic or randomized mistake bound as the accumulating loss of the learner through the entire game, or simply as the learner’s loss through the entire game.
Decision Trees and the Littlestone Dimension.
In this paper, a tree refers to a finite full rooted ordered binary tree (that is, a rooted binary tree where each node which is not a leaf has a left child and a right child), equipped with the following information:
Each internal node is associated with an instance . 2. 2.
For every internal node , the left outgoing edge is associated with the label [math], and the right outgoing edge is associated with the label .
We stress that by default, the trees we consider are finite and their vertices are labeled. Whenever we consider infinite trees or unlabeled trees, we specifically mention these attributes.
The tree is directed from the root towards the leaves.
A prefix of the tree is any path that starts at the root. In this paper, a path is defined by a sequence of consecutive vertices. If a path is not empty, we may refer it by the sequence of consecutive edges corresponding with the sequence of consecutive vertices defining it. A prefix defines a sequence of examples in a natural way: for every , is the instance corresponding to the node , and is the label corresponding to the edge . A prefix is called maximal if it is maximal with respect to containment, that is, there is no prefix in the tree that strictly contains it. This is equivalent to requiring that be a leaf. A maximal prefix is called a branch, and the set of branches of is denoted by . The length of a prefix is the number of edges in it (so, the length is equal to the size of the corresponding sequence of examples).
A prefix in the tree is said to be realizable by if the corresponding sequence of examples is realizable by . A tree is shattered by if all branches in are realizable by . The Littlestone dimension of an hypothesis class , denoted by , is the maximal depth of a complete (also known as perfect, or balanced) binary tree (that is, a tree in which all branches have the same depth) shattered by if , and when . If the maximum does not exist, then .
Littlestone Dimension Optimal Deterministic Mistake Bound.
In his seminal work from 1988, Nick Littlestone proved that the optimal mistake bound of a deterministic learner is characterized by the Littlestone dimension:
Theorem 4.1** (Optimal Deterministic Mistake Bound [Lit88]).**
*Let be an hypothesis class. Then, is online learnable if and only if . Further, the optimal deterministic mistake bound satisfies . *
Doob’s Exposure Martingales.
Let . Consider the random variable , where is sampled uniformly at random. Define a sequence , each defined by (so ). The sequence is called an exposure martingale. It is well-known that an exposure martingale is indeed a martingale [Doo53].
5 Randomized Littlestone Dimension and Optimal Expected Mistake Bound
In this section we study the randomized Littlestone dimension. We shall define the randomized Littlestone dimension and prove that it characterizes the optimal randomized mistake bound exactly.
The randomized Littlestone dimension is defined using trees, which correspond to strategies of the adversary. We study a special class of trees, quasi-balanced trees, in Section 6.1. Such trees define optimal strategies for the adversary (more details on such strategies are found in the full version of this paper [FHMM22]). Several other applications of quasi-balanced trees are presented in the full version of this paper [FHMM22]; more applications are found throughout the paper.
The first main contribution of this paper is a characterization of the optimal randomized mistake bound in terms of a combinatorial parameter we call the randomized Littlestone dimension and denote by .
We define using a natural distribution on the branches of trees (a branch is a root-to-leaf path). Given a tree , a random branch is chosen by starting at the root, and at each step, picking an edge leaving the current vertex uniformly at random, until reaching a leaf. We denote the expected length of a random branch by . It is given explicitly by the formula
[TABLE]
where is the set of branches of . If we think of a random branch as a distribution over , then is its entropy.
It is convenient to define the length of the empty branch to be . With this convention, the expected branch length in satisfies the recursion
[TABLE]
where are the subtrees of the root of , which are empty when is a leaf.
Definition 5.1** (Randomized Littlestone Dimension).**
Let be an hypothesis class. The randomized Littlestone dimension of , denoted by , is defined by
[TABLE]
In the special case when , define .
To compare with , let us consider the following equivalent way of defining :
[TABLE]
where is the minimum length of a branch in . Thus, the difference is that in we take the expected depth rather than the minimal depth, and multiply by a factor of .
Theorem 5.2** (Optimal Randomized Mistake Bound).**
Let be an hypothesis class. Then,
[TABLE]
We prove the theorem in Subsection 5.1 using randomized , a randomized adaptation of Littlestone’s classical algorithm. This shows that the infimum in Equation (2) is realized by a minimizer.
5.1 Proof of Characterization
The case holds by definition. Therefore we assume that . The lower bound “” boils down to the following lemma:
Lemma 5.3**.**
Let be an hypothesis class, and let be a finite tree which is shattered by . Then, for every learning rule there exists a realizable sequence so that . Moreover, there exists such a sequence which corresponds to one of the branches of .
Proof.
The proof is given by a simple probabilistic argument. Suppose that we pick a random branch in the tree according to the random branch distribution: begin at the root, pick a random child of the root uniformly at random, and recursively pick a random branch in the corresponding subtree. Consider the random variable
[TABLE]
where is the sequence of examples corresponding to a random branch drawn as above. It suffices to show that . We prove this by induction on the depth of .
In the base case, is a single leaf, and there are no internal nodes. Hence is always the empty sequence, and , as required.
For the induction step, let and be the left and right subtrees of , respectively. The expected loss of on the first example in is , because the label is chosen uniformly at random, independently of the learner’s prediction (formally, for all ). Therefore, by linearity of expectation,
[TABLE]
as required. ∎
By applying Lemma 5.3 on every shattered tree and taking the supremum, we conclude the lower bound:
Corollary 5.4** (Lower bound).**
For every hypothesis class it holds that .
We now turn to prove the upper bound “”. This is achieved via the learning rule, described in Figure 2.
We begin with the following useful property of :
Observation 5.5**.**
Let be a non-empty hypothesis class. Then,
[TABLE]
Proof.
Observation 5.5 follows from Equation (3): let denote the set of trees that are shattered by , and for , let denote the set of trees that are shattered by whose root is labeled by . Then,
[TABLE]
By Equation (3),
[TABLE]
which finishes the proof. ∎
Notice that the classical Littlestone dimension satisfies a similar recursion:
[TABLE]
The following lemma is the crux of the analysis: it guides the choice of the prediction in each round.
Lemma 5.6** (Optimal prediction for each round).**
Let be an hypothesis class, and let . Then there exists so that
[TABLE]
Proof of Lemma 5.6.
If then the lemma is trivial. Therefore we assume that . Assume first that . If , then by choosing and applying the fact that if we get
[TABLE]
as desired. The case is treated similarly.
It remains to handle the case when . Set
[TABLE]
By assumption, , and also
[TABLE]
∎
Lemma 5.7** (Upper bound).**
Let be an hypothesis class. Then the learner described in Figure 2 has expected mistake bound
[TABLE]
for every realizable input sequence .
Proof.
The proof is by induction on the length of the input sequence. Let be a realizable sequence. In the base case we have . For the induction step, assume that , and let be the input sequence without the first example. In the first round, the learner predicts as defined in step 2 of . Thus, the learner’s expected accumulated loss on is
[TABLE]
By the induction hypothesis we have
[TABLE]
Also, by Lemma 5.6 it holds that and , which is equivalent to
[TABLE]
Therefore, overall we get that
[TABLE]
as required. ∎
6 Quasi-balanced Trees
6.1 Definition and Basic Properties
The classical definition of the Littlestone dimension of a class is the maximum depth of a balanced (or complete) shattered tree. In contrast, the randomized Littlestone dimension is defined via quantifying over all shattered trees. Further, in the deterministic case, balanced trees naturally describe optimal deterministic strategies for the adversary which force any learner to make a mistake on every example along a branch of the tree.
It is therefore natural to ask whether there is a type of shattered trees, analogous to balanced trees, which can be used to define the randomized Littlestone dimension. In this subsection, we show that such an analog exists: a type of trees which we call quasi-balanced; roughly speaking, quasi-balanced trees can be seen as a fractional relaxation of balanced trees. We further use this section to prove some useful properties of these trees, which will be used later on.
Informally, quasi-balanced trees are balanced under some weight function defined on the edges. To formally define quasi-balanced trees, we need to define weight functions for trees.
Let be a non-empty tree with edge set . Let be the set of all functions , such that for every internal node with outgoing edges it holds that . Each function in is called a weight function for .
For every branch defined by a sequence of consecutive edges, define the weight of the branch with respect to by .
The expected weight of a random branch is always half the expected length of a random branch, as a simple inductive argument shows.
Lemma 6.1**.**
For every non-empty tree and every weight function , the expected weight of a random branch is .
Proof.
The proof is by induction on the depth of the tree. If is a leaf then the expected weight of a random branch is . If is not a leaf, let be the edges emanating from the root, and let be the corresponding subtrees. Applying the inductive hypothesis, the expected weight of a random branch in under is
[TABLE]
using and Equation (3). ∎
This lemma prompts the following definition.
Definition 6.2**.**
A tree is quasi-balanced if it is non-empty and there is a weight function under which all branches have weight .
We call the weight of the tree, and denote it by .
Lemma 6.3**.**
If a tree is quasi-balanced then there is a unique weight function under which all branches have the same weight. Explicitly, if is a subtree of whose root is connected via edges to the subtrees , then
[TABLE]
Proof.
The trees are necessarily quasi-balanced, and in particular
[TABLE]
Since , we can solve for , obtaining the claimed formula. ∎
Quasi-balanced trees are a generalization of balanced trees: every tree which is balanced is also quasi-balanced with weight , where is the depth of . This weight is realized by the (unique) constant weight function that gives weight to all edges. The family of quasi-balanced trees is, however, much broader than the family of balanced trees.
Recall the definition of the randomized Littlestone dimension of the class :
[TABLE]
It turns out that in this definition, it suffices to take the supremum only over quasi-balanced trees. This will be easier to see through the characterization of quasi-balanced trees as monotone trees.
Definition 6.4** (Monotone Trees).**
A non-empty tree is weakly monotone if
[TABLE]
where and are the subtrees rooted at the children of the root of . A tree is monotone if it is non-empty and all of its subtrees are weakly monotone.
It is not hard to see that non-monotone trees need not be considered when computing the randomized Littlestone dimension.
Lemma 6.5**.**
For any non-empty hypothesis class ,
[TABLE]
Proof.
Consider a tree shattered by which is not monotone. Then there exists a vertex such that , where is a tree rooted at a child of . If we replace the subtree rooted at with the subtree , we get a tree which is also shattered by , and has higher expected branch length.
Repeating this process finitely many times, for each tree shattered by we obtain a monotone tree shattered by satisfying , and the lemma follows. ∎
The following theorem asserts that monotone and quasi-balanced trees are indeed equivalent.
Theorem 6.6**.**
*A tree is quasi-balanced if and only if it is monotone. *
Corollary 6.7**.**
For any non-empty hypothesis class ,
[TABLE]
To prove Theorem 6.6, we use the following simple observation.
Observation 6.8**.**
Let be a non-empty tree. Then is weakly monotone if and only if , where and are the subtrees rooted at the children of the root of .
Proof.
Equation (3) states that , and so is equivalent to . Similarly, is equivalent to . Hence is weakly monotone iff . ∎
Proof of Theorem 6.6.
An empty tree is neither quasi-balanced nor monotone. Suppose therefore that we are given a non-empty tree . We prove the equivalence by proving both implications separately.
Monotone Quasi-balanced.
The proof is by induction on the depth of the tree. A tree of depth [math] (the base case) is quasi-balanced with weight . For the induction step, let be the subtrees rooted at the root’s children. They are clearly monotone, and so by induction, there are weight functions and under which all branches in have weight and all branches in have weight .
Let be the edges connecting the root of to the roots of , respectively. Define a weight function by defining if , if ,
[TABLE]
Clearly . Observation 6.8 implies that , and so indeed . Since , the weight function shows that is quasi-balanced.
Quasi-balanced Monotone.
The proof is by induction on the depth of the tree. A tree of depth [math] is monotone. For the induction step, we first observe that every proper subtree of is quasi-balanced, and so monotone by the inductive hypothesis. Hence it suffices to show that is weakly monotone.
Let be the unique weight function for under which each branch has weight . Let be the edges connecting the root of to the two subtrees . According to Lemma 6.3, the weights of these edges are
[TABLE]
Since the weights are non-negative, we deduce that , and so . We conclude that is weakly monotone by Observation 6.8. ∎
6.2 A Concentration Lemma for Quasi-Balanced Trees
Another interesting property of quasi-balanced trees is that the length of a random branch concentrates around its expectation. This property will be important for deriving tight bounds in Section 9.
Proposition 6.9** (Concentration of branch lengths).**
Let be a quasi-balanced tree, and let be the length of a random branch. Then for any ,
[TABLE]
Proof.
If is a single leaf then the result trivially holds since there is a single random branch. Therefore we can assume that is not a single leaf, and in particular, .
Let be an infinite sequence of random coin tosses. We can choose a random branch of as follows. Let be the root of . For , if is not a leaf, then is obtained by following the edge labeled . Otherwise, we define . The resulting random branch has exactly the same distribution that we have been considering so far.
Let be the expected length of the branch given . This is an exposure martingale, as defined in Section 4.
In order to apply Azuma’s inequality, we need to bound the random difference . If is a leaf, then . Otherwise, let be the subtree rooted at , and let be the subtrees rooted at the children of . Thus is either or , depending on the value of . Moreover, is the average of these two values.
Theorem 6.6 shows that is weakly monotone, and so Observation 6.8 shows that . Consequently,
[TABLE]
The definition of implies that for all . In particular, if then . Applying Azuma’s inequality and using , it follows that
[TABLE]
where the final inequality uses .
The definition of also implies that whenever . In particular, if then . Therefore
[TABLE]
using as before. ∎
7 Bounded Horizon
So far we have not put any restrictions on the number of rounds. However, in many circumstances we are interested in the online learning game when the number of rounds is bounded. We model this by assuming that the learner knows an upper bound on the number of rounds. We define to be the optimal randomized mistake bound when the number of rounds is at most .
We can generalize Theorem 5.2 to this setting. The required notion of randomized Littlestone dimension is
[TABLE]
The bounded randomized Littlestone dimension gives the precise mistake bound in this setting.
Theorem 7.1** (Optimal Randomized Mistake Bound with Finite Horizon).**
Let be an hypothesis class, and let . Then,
[TABLE]
We prove Theorem 7.1 in Section 7.1. This theorem immediately suggests the following questions:
How many rounds are needed in order for the adversary to guarantee that the loss of the learner is at least ?
We prove in Section 7.2 that rounds always suffice, and rounds suffice as long as is small enough. 2. 2.
What can we say about the loss of the learner when there are fewer than rounds?
A trivial upper bound on is . In Section 7.3 we show that this bound is nearly optimal when .
The proofs of these results use concentration bounds on the depth of quasi-balanced trees, which we prove in Section 6.2.
7.1 Proof of Theorem 7.1
In this section we indicate how to generalize the proof of Theorem 5.2 to the finite horizon setting, proving Theorem 7.1.
The lower bound follows directly from the statement of Lemma 5.3, since the length of is at most .
For the upper bound, we use a straightforward modification of algorithm , which appears in Figure 3.
We start by extending Observation 5.5: if is a non-empty hypothesis class and then
[TABLE]
The proof is identical. Since there are only finitely many unlabeled trees of depth at most , we can replace the supremum with a maximum.
The next step is to generalize Lemma 5.6, which now states that for every hypothesis class , instance , and , there exists so that
[TABLE]
The proof is identical, using the generalized Observation 5.5.
Finally, we prove the following generalization of Lemma 5.7: for every hypothesis class , any parameter , and any realizable input sequence of length at most ,
[TABLE]
The proof is identical, using the generalized Lemma 5.6.
7.2 Approaching
As a simple consequence of the concentration bound proved in Proposition 6.9, we show that we can approach using relatively shallow trees, quantified as follows.
Proposition 7.2**.**
Let be a non-empty hypothesis class with finite randomized Littlestone dimension .
For every there is a tree shattered by satisfying whose depth is at most
[TABLE]
This means that the adversary can force the learner to suffer a loss of after only rounds. In contrast, at least rounds are clearly needed, since a learner who predicts at each round suffers a loss of after rounds.
We prove Proposition 7.2 via the following technical estimate.
Lemma 7.3**.**
Let be a monotone tree, and let result from truncating it to the first levels (all branches in have length at most ). If then
[TABLE]
Proof.
Let be the length of a random branch of . Using , we can express the difference between and explicitly:
[TABLE]
Applying Proposition 6.9, we deduce that the difference is at most
[TABLE]
If then
[TABLE]
using the well-known Gaussian tail bound,
[TABLE]
to bound the first integral.
If then
[TABLE]
We can now prove Proposition 7.2.
Proof of Proposition 7.2.
Applying Lemma 6.5, we can find a monotone tree shattered by such that . Let
[TABLE]
Lemma 7.3 implies that , and so . ∎
7.3 Mistake Bound for Few Rounds
Another truncation argument allows us to estimate for small .
Proposition 7.4**.**
Let be a non-empty hypothesis class with finite randomized Littlestone dimension .
If then .
If then
[TABLE]
Furthermore, if then
[TABLE]
and if then
[TABLE]
Proof.
A learner that always predicts suffers a loss of exactly each round, showing that for each . In contrast, if is a tree shattered by then Theorem 7.1 shows that , and we will use this to give lower bounds on .
Suppose first that . Our characterization shows that . If is close enough to then (since is an integer), and so is a complete tree of depth . This shows that .
In order to prove the remaining results, suppose that , and consider a tree shattered by satisfying . Proposition 6.9 shows that a random branch of has depth with probability at least 1-\exp\bigl{(}-\frac{(E_{T}-\mathbf{T})^{2}}{4E_{T}}\bigr{)}, and so
[TABLE]
where the limit is taken along a sequence of trees shattered by and satisfying .
If then this gives
[TABLE]
If then
[TABLE]
Finally, if we only assume that then
[TABLE]
8 Mistake Bounds in the -Realizable Setting
So far we have considered online learning when the adversary is restricted to choose labels which are consistent with one of the hypotheses in the hypothesis class, a setting known as the realizable setting. This is a quite restrictive assumption, and there are many ways to relax it.
In this section we concentrate on the -realizable setting, in which the answers of the adversary are consistent with one of the hypotheses in the class up to at most mistakes. Our goal is to characterize the optimal mistake bounds in this setting, for both deterministic and randomized learners, generalizing Theorems 4.1 and 5.2. Our characterizations are based on -shattered trees, in which each branch is consistent with one of the hypotheses in the class up to at most mistakes.
If all instances in a sequence of examples are distinct, then the sequence is -realizable by if and only if it is realizable by the -expansion of , consisting of all hypotheses which disagree with some hypothesis on at most instances. However, this need not be the case. For example, the sequence is -realizable by the hypothesis class consisting of all constant functions.
Nevertheless, the arguments in this section are very similar to their counterparts in the realizable setting.
8.1 Weighted Hypothesis Classes
While we are interested mainly in the -realizable setting, we consider a more general setting in which the number of allowed mistakes can depend on the hypothesis. This will be useful in the subsequent proofs.
A weighted hypothesis class is a collection of pairs , where is an hypothesis and is the allowed number of mistakes (possibly zero). Furthermore, all hypotheses are distinct (that is, cannot contain two different pairs ). An input sequence is realizable by a weighted hypothesis class if there exists such that for at most many examples in the sequence. A tree is shattered by if each of its branches is realized by .
Given an hypothesis class , a learning rule which observes the labeled example can restrict itself to . The corresponding operation for weighted hypothesis classes is
[TABLE]
In words, we decrease the allowed number of mistakes for each hypothesis inconsistent with the given example , removing hypotheses which has zero mistakes left.
For every weighted hypothesis class , we define its Littlestone dimension and its randomized Littlestone dimension by
[TABLE]
where the supremum is taken over all trees shattered by . As in the realizable setting, we define for convenience.
Our main results in this section extend Theorems 4.1 and 5.2 to this more general setting.
Theorem 8.1** (Optimal Deterministic Mistake Bound).**
Let be a weighted hypothesis class. Then,
[TABLE]
Theorem 8.2** (Optimal Randomized Mistake Bound).**
Let be a weighted hypothesis class. Then,
[TABLE]
We prove these theorems in the following subsections, making use of the following fundamental observation, which follows directly from the definitions:
Observation 8.3**.**
Let be a weighted hypothesis class. The sequence is realizable by iff the sequence is realizable by .
Similarly, let is a tree whose root is labeled by , and let be the subtrees rooted at the children of the root. Then is realizable by iff is realizable by and is realizable by .
The -realizable setting.
Let be an hypothesis class, and let . A sequence of examples is -realizable by if there exists such that for at most indices . We denote the corresponding mistake bounds by . These are defined just as in the realizable setting, the only difference being that the sequence of examples provided by the adversary need only be -realizable by .
We say that a tree is -shattered by if every branch is -realizable by . The corresponding deterministic and randomized -Littlestone dimension of a class are
[TABLE]
If we define , then a sequence of examples is -realizable by if it is realizable by . In other words, the -realizable setting is a special case of weighted hypothesis classes, where all weights are equal to . Therefore we immediately conclude the following theorems, by applying the preceding theorems to :
Theorem 8.4** (Optimal Deterministic Mistake Bound).**
Let be an hypothesis class, and let . Then,
[TABLE]
Theorem 8.5** (Optimal Randomized Mistake Bound).**
Let be an hypothesis class, and let . Then,
[TABLE]
Using the classic lower bounds of [LW94, BDPSS09] and recent results of [ABED*+*21], we can bound the optimal mistake bound in terms of the realizable Littlestone dimension:
Theorem 8.6**.**
Let be an hypothesis class with at least two hypotheses, and let . Then,
[TABLE]
We prove this result in Section 8.4. Note that since and differ by at most a constant factor, the theorem still holds if we replace by .
Using the experts algorithm of [KvE15], we can construct an algorithm which works in the adaptive setting, that is, without knowledge of :
Theorem 8.7**.**
Let be an hypothesis class. There is an algorithm such that for every input sequence which is -realizable by ,
[TABLE]
Furthermore, is adaptive, that is, it has no knowledge of .
We describe and analyze the algorithm in Section 8.5.
8.2 Proof of Optimal Deterministic Mistake Bound
The case holds by definition. Therefore we assume that . The lower bound “” boils down to the following lemma:
Lemma 8.8**.**
Let be a weighted hypothesis class, and let be a finite tree which is shattered by . Then, for every deterministic learning rule there exists a realizable sequence so that . Furthermore, corresponds to one of the branches in .
Proof.
We construct the sequence by traversing , starting at the root . At step , we send the instance labelling . If the learner predicts , we set the true label to , and let be the vertex obtained from by following the leaf labeled . We stop once the process reaches a leaf.
By construction, corresponds to one of the branches of , and the number of mistakes is . Since is shattered by , then is realizable by . ∎
By applying the lemma on every shattered tree and taking the supremum, we conclude the lower bound:
Corollary 8.9** (Lower bound).**
For every weighted hypothesis class it holds that .
We now turn to prove the upper bound “”. This is achieved via the learning rule, depicted in Figure 4.
Lemma 8.10** (Upper bound).**
Let be a non-empty weighted hypothesis class. The learner described in Figure 4 has the mistake bound
[TABLE]
for every input sequence realizable by .
Proof.
We will show that each time that makes a mistake, the Littlestone dimension drops by at least . That is, if then . Since the Littlestone dimension is always non-negative, it follows that makes at most mistakes.
Suppose that yet . The choice of shows that . This is, however, impossible. Indeed, take trees shattering with . Observation 8.3 shows that the tree whose root is labeled and in which are the subtrees of the root’s children is shattered by . Since , we reach a contradiction. ∎
8.3 Proof of Optimal Randomized Mistake Bound
The proof of the optimal mistake bound in the randomized setting, Theorem 8.2, is very similar to the proof of its counterpart in the realizable setting, Theorem 5.2.
The proof of the lower bound “” is virtually identical to the proof of Lemma 5.3.
The proof of the upper bound “” uses , the weighted counterpart of , which appears in Figure 5. The proof of Lemma 5.7 extends, with virtually no changes, to show that for every input sequence realizable by .
8.4 Explicit Bounds in Terms of Littlestone Dimension
Here we prove Theorem 8.6, which bounds in terms of and (or ). In the proof, we use the notation for the optimal mistake bound in the -realizable setting when the number of rounds is bounded by , and the notation for the optimal mistake bound when the number of rounds is bounded by , but there is no limitation on the number of mistakes made by the best hypothesis in .
We first prove the upper bound. [ABED*+*21] have shown that, for any time horizon , we always have . By Proposition 7.2, time horizon suffices to guarantee . Plugging this time horizon into the result of [ABED*+*21] reveals that
[TABLE]
Solving this quadratic inequality in yields the upper bound claimed in the theorem.
We now turn to prove the lower bound. We consider two cases. If , it suffices to prove a lower bound of . The lower bound of [LW94] establishes that. In the complementary case, suppose that . Therefore, we only need to prove a lower bound of . This follows from the following adaptation of the classic regret bound of [BDPSS09]. They showed that , where is the minimal number of mistakes made by a best hypothesis . To adapt this bound to our setting, first play the game for rounds, forcing a loss of at least on the learner. Now, as we prove in Theorem 9.3,111111We prove this result in the setting of prediction with expert advice, but it holds for general hypothesis classes (as long as the domain is non-empty). the adversary can further force the learner a loss arbitrarily close to , using an input sequence which is -realizable by . Overall, the input sequence is -realizable by , and we get the desired lower bound .
8.5 Adapting to
This section presents our proof of Theorem 8.7, showing that it is possible to adapt to the value of without sacrificing too significantly in the expected mistake bound.
The adaptive technique we propose uses an experts algorithm of [KvE15] named , with experts defined by the optimal randomized algorithm for the -realizable setting, for all values of (including ).
The experts algorithm accepts an input sequence and a list of learners , each with an associated weight . The weights should form a probability distribution. With an appropriate choice of parameters, has the following guarantee [KvE15, Theorem 3]:
[TABLE]
where is an uncentered variance term given by
[TABLE]
Since both absolute values are in the range , we have
[TABLE]
For any given , if , then we have , so that (8) implies
[TABLE]
This inequality trivially holds as well in the case due to the first term on the right hand side. Moreover, this inequality further implies
[TABLE]
To see this, note that were it not the case, we could upper bound each and on the right hand side by for some large constant , making the right hand side strictly less than : a contradiction. Plugging in this upper bound on into the term and simplifying with elementary inequalities reveals
[TABLE]
This is a quadratic inequality in . Solving the quadratic for the range of where the inequality holds, we have
[TABLE]
Since this holds for any , we conclude that
[TABLE]
We instantiate with algorithm of Figure 5. Namely, for every , we let be the instantiation of with . We use the weights . Since , they indeed constitute a probability distribution. Since achieves the optimal mistake bound (see Section 8.3), Eq. (9) shows that if is -realizable by then
[TABLE]
Since , the term \log\bigl{(}(k^{\star}+1)\texttt{M}^{\star}(\mathcal{H},k^{\star})\bigr{)} can be swallowed by the preceding term.
9 Prediction using Expert Advice
In this section, we consider the problem of prediction using expert advice, which was raised in [Vov90, LW94]. Specifically, we consider the -realizable setting, which was suggested in [CBFHW96, CBFH*+*97] and further studied in [ALW06, MS10, BP19].
The problem concerns a repeated game which has the same flavor as the online learning game of Section 4. The game is between a learner and an adversary. Additionally, there are experts. Each round in the game proceeds as follows:
- (i)
The experts present predictions . 2. (ii)
The learner predicts a value . 3. (iii)
The adversary reveals the true answer , and the learner suffers the loss .
The adversary must choose the answers so that at least one of the experts makes at most mistakes. That is, there must exist an expert such that for at most many indices . We call such an adversary -consistent.
The goal is to determine the optimal loss of the learner as a function of and . We denote the optimal loss of the learner by , and the optimal loss when the learner is constrained to output predictions in by .
The game underlying prediction using expert advice is quite similar to the online learning game. In fact, we can relate the two.
Let , and consider the hypothesis class on the domain consisting of the projection functions . We can simulate the game of prediction using expert advice by the online learning game as follows: whenever the experts predict , the adversary sends the instance . The adversary in the original game is -consistent if and only if the sequence is -realizable by .
This simulation goes both ways, and so the two games are actually equivalent. The upshot is that we can express and in terms of quantities we have already considered:
[TABLE]
The equivalence above shows that is the “hardest” hypothesis class of size , in the sense that it maximizes both and over all hypothesis classes of size . Indeed, and are equal to the optimal loss in the game of prediction using expert advice when the answers of the experts must belong to , where has domain .
Bounded horizon.
Prediction using expert advice is often considered when the number of rounds is bounded. Let be the optimal loss of the learner when the number of rounds is .
Clearly . In view of Theorem 7.1, Proposition 7.2 shows that already for . In contrast, since a learner can always guarantee a loss of at most per round by predicting , we have , and so requires .
(The deterministic case is not interesting, since trivially .)
9.1 Optimal Mistake Bounds
For every and , let
[TABLE]
The value of plays a central role in the problem of prediction using expert advice: [CBFHW96] showed that using the Binomial Weights learning rule, and complemented this with an asymptotically matching lower bound . The lower bound is proved by constructing a -covering code of size that simulates the experts. When is fixed and is large enough, it can be further improved to , as shown in [CBFHW96].
The paper [CBFHW96] leaves open the problem of determining whether for some universal constant . [CBFH*+*97] showed that whenever or .121212More precisely, they showed that . Together with the bound of [LW94], this implies that whenever or . [ALW06] showed that for large enough (as a function of ), . [BP19] showed that for , even in the multiclass setting where the experts’ predictions are chosen from some finite set . In this section, we remove any assumptions on , proving the following theorem:
Theorem 9.1**.**
Let and . Then
[TABLE]
The error term is tight for :
Theorem 9.2**.**
Let . Then
[TABLE]
All of our bounds are attained using the randomized -Littlestone dimension of . Note that as a special case of Theorem 8.6, one can also derive the bounds , using . The upper bound was proved by [CBFH*+*97]. We prove the upper bound in Section 9.2, and the lower bound in Section 9.3. In the full version of the paper, we use our techniques to determine exactly.
All results we stated so far concern . The case is different, and much simpler:
Theorem 9.3**.**
Let . Then
[TABLE]
Proof.
According to the definition, is the maximum such that . Since whereas , we see that .
The complete tree of depth , labeled arbitrarily, is -shattered by . In contrast, a tree of depth cannot be -shattered by , since there exists a branch on which the unique hypothesis makes mistakes. Therefore .
For the randomized case,it is convenient to consider an optimal infinite tree such that (proof of existence of can be found in the full version of the paper [FHMM22]). Denote the unique hypothesis in by . By possibly switching the order of children, we can assume that all vertices in are labeled by an instance such that . We can then identify vertices of with binary strings.
Since is optimal, it contains all strings which contain at most many s. A string is a leaf it it contains exactly many s and it ends with . The length of a random branch has the distribution of a sum of many random variables, and so . ∎
In contrast, [LW94] shows that for , highlighting the difference between and . This immediately implies the following corollary, which will be useful in the sequel:
Corollary 9.4**.**
Let and . Then .
Proof.
Clearly . Theorem 9.6 shows that , which is at least by the result of [LW94]. ∎
9.2 Proof of the Upper Bound on
We start by proving a probabilistic version of the sphere packing bound for covering codes [CHLL97].
Lemma 9.5**.**
Let be a finite hypothesis class of size . Let , and let be a tree whose minimum depth is at least .
Let be the random prefix of length , consisting of the first steps in a random branch of . The probability that is -realizable by is at most
[TABLE]
Proof.
For each hypothesis and set of indices , the probability that for all indices in and for all indices outside of is .
The sequence is -realizable by if the event above happens for some and some of size at most . Applying the union bound, we get that the probability is at most . ∎
As a warm-up, we use this lemma together with the -Littlestone dimension to reprove the upper bound , first proved in [CBFHW96].
Theorem 9.6**.**
Let and . Then .
Proof.
Since , it suffices to bound .
Let be a tree satisfying which is -shattered by . A random prefix of length is -realizable by , and so by Lemma 9.5. Taking the logarithm, we deduce that by the definition of . ∎
We now prove Theorem 9.1. The main tools are concentration of the random branch length in quasi-balanced trees (Lemma 6.9), and the following lemma.
Lemma 9.7**.**
Let be a finite hypothesis class of size . Let , and let be a tree of minimum depth at least , where . The probability that a random prefix of length is -realizable by is at most
[TABLE]
Furthermore, if for some constant then the probability is at most
[TABLE]
where is a constant depending only on .
The proof of this lemma will require some elementary estimates on binomial coefficients, summarized in the following technical lemma.
Lemma 9.8**.**
Let and . Then
[TABLE]
If furthermore and then
[TABLE]
We prove this lemma in Subsection 9.2.1.
Proof of Lemma 9.7.
We start by observing that
[TABLE]
Indeed, the maximality of shows that
[TABLE]
from which Eq. (10) immediately follows.
Denote by the probability we wish to bound. Lemma 9.5 shows that
[TABLE]
using Eq. (10). It remains to estimate the ratio using Lemma 9.8.
We start by proving the “furthermore” part. By assumption, we have . Applying Lemma 9.8, we deduce that
[TABLE]
Since
[TABLE]
this completes the proof of the “furthermore” part.
In order to prove the main part of the lemma, we distinguish between two cases. If then the “furthermore” bound shows that
[TABLE]
where . Since , we have , completing the proof in this case.
Otherwise, . In this case, noting that by Corollary 9.4, we apply the “furthermore” part of Lemma 9.8 to obtain
[TABLE]
We can now prove the upper bound on . The idea is simple. Let be a tree which is -shattered by . Using Corollary 6.7 , we can assume that is quasi-balanced, and so the length of a random branch is concentrated around .
This implies that realizes almost all sequences of size . These sequences are -realized by , and we obtain an upper bound on via Lemma 9.7.
Proof of Theorem 9.1.
Since , we bound the latter. It is convenient (although not necessary) to assume that there is an infinite tree which is -shattered by and satisfies 131313This assumption is proved to be true in the full version of this paper [FHMM22]. Furthermore, is monotone, and so Proposition 6.9 applies to it (while the proposition is formulated for finite quasi-balanced trees, the proof actually directly uses monotonicity, and is valid for infinite trees).
In order to bound , we will show that for small enough , the assumption leads to a contradiction.
Extend arbitrarily to a tree of minimum depth , and let be a random prefix of of length . If lies completely within then it is -realizable by , and so
[TABLE]
The probability that lies within is precisely the probability that a random branch of has length at least . Since we assume that , this probability is at least by Proposition 6.9, and so at least .
In contrast, the probability that is -realizable by is at most by Lemma 9.7. Therefore
[TABLE]
Let . As , the right-hand side tends to [math], and in particular, we obtain a contradiction for some constant .
It follows that for , and so
[TABLE]
9.2.1 Proof of Technical Estimate
In this section we complete the proof of Theorem 9.1 by proving Lemma 9.8.
We start with estimates on the ratio of individual binomial coefficients.
Lemma 9.9**.**
Let and . Then
[TABLE]
If furthermore and then
[TABLE]
Proof.
We can calculate the ratio between the binomials explicitly:
[TABLE]
Applying the well-known estimate , we obtain
[TABLE]
and so
[TABLE]
Now suppose that and . For we have
[TABLE]
Since for , we can improve the estimate on :
[TABLE]
We can now prove Lemma 9.8.
Proof of Lemma 9.8.
The ratio between and is clearly at most , where is the ratio between the binomials in Lemma 9.9.
If we only assume that and , then Lemma 9.9 states that
[TABLE]
which is clearly monotone increasing in . Therefore
[TABLE]
If we furthermore assume that and , then Lemma 9.9 states that
[TABLE]
The derivative of the upper bound with respect to is
[TABLE]
since . Therefore the upper bound is maximized at , and we conclude that
[TABLE]
Since , we can further estimate
[TABLE]
9.3 Lower bounding
We prove Theorem 9.2 by applying the lower bound of Theorem 8.6. In the full version of this paper [FHMM22], we also show how our techniques can be used to determine the exact value of . Concretely, we are able to identify the optimal shattered tree and to compute its expected branch length.
Proof of Theorem 9.2.
Since , Theorem 8.6 implies that
[TABLE]
On the other hand, it is easy to see that . Indeed, if then
[TABLE]
with equality if and only if . Therefore . ∎
10 Open Questions
Our work naturally raises many directions for future research (more can be found in the full version paper [FHMM22]).
Adaptive algorithms.
Algorithm gives the optimal mistake bound, but requires knowledge of . Theorem 8.7 gives an algorithm which doesn’t require knowledge of , and has a regret bound of (this is the loss beyond ). What is the optimal regret bound?
Quantitative bounds.
The first part in Theorem 2.6 asserts that . It will be interesting to get quantitative bounds on the second-order term in terms of . By the second part of Theorem 2.6 we know that in some cases () it is . Does an upper bound of
[TABLE]
hold for all ?
In addition, it will be interesting to find explicit bounds on ; by Theorem 2.5, we know that when then141414The upper bound is also given in [CBFH*+*97].
[TABLE]
How about for other values of ? Brânzei and Peres [BP19] have the state-of-the-art bounds in the regime , but we are not aware of any results in other regimes, e.g. when . As a matter of fact, to the best of our knowledge, even the leading asymptotic terms in the regime were unknown prior to this work (we included them in the full version of this work [FHMM22]).
Proper predictions and repeated game playing.
Consider the prediction using the expert advice problem, when the learner is restricted to predict with a convex combination of the experts. That is, at the beginning of each round (before seeing the advice of the experts), the learner picks a convex combination of the experts and predicts accordingly. What is the optimal expected number of mistakes in this game?151515We answer this question only for the case in the full version of this paper [FHMM22].
The optimal algorithm for this problem was identified in [AWY08]. We comment that this game can also be presented in the language of game theory: assume a repeated zero-sum game with values, where each round is played as follows: player (i) chooses a (mixed) strategy and reveals it to player (ii), who then replies with a strategy of his own. What is the optimal accumulated payoff that player (i) can guarantee provided that she has strategies and that the sequence of strategies chosen by player (ii) is such that player (i) has a pure strategy that loses to at most of them? Proper predictions in the prediction with expert advice setting are equivalent to mixed strategies here.
Acknowledgments
Shay Moran is a Robert J. Shillman Fellow; he acknowledges support by ISF grant 1225/20, by BSF grant 2018385, by an Azrieli Faculty Fellowship, by Israel PBC-VATAT, by the Technion Center for Machine Learning and Intelligent Systems (MLIS), and by the the European Union (ERC, GENERALIZATION, 101039692). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1[ABED + 21] Noga Alon, Omri Ben-Eliezer, Yuval Dagan, Shay Moran, Moni Naor, and Eylon Yogev. Adversarial laws of large numbers and optimal regret in online classification. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing , pages 447–455, 2021.
- 2[ABL + 22] Noga Alon, Mark Bun, Roi Livni, Maryanthe Malliaris, and Shay Moran. Private and online learnability are equivalent. J. ACM , 69(4):28:1–28:34, 2022. doi:10.1145/3526074 . · doi ↗
- 3[AL 99] Peter Auer and Philip M Long. Structural results about on-line learning models with and without queries. Machine Learning , 36(3):147–181, 1999.
- 4[ALW 06] Jacob Abernethy, John Langford, and Manfred K Warmuth. Continuous experts and the binning algorithm. In International Conference on Computational Learning Theory , pages 544–558. Springer, 2006.
- 5[AWY 08] Jacob Abernethy, Manfred K Warmuth, and Joel Yellin. Optimal strategies from random walks. In Proceedings of The 21st Annual Conference on Learning Theory , pages 437–446, 2008.
- 6[BCD + 22] Nataly Brukhim, Daniel Carmon, Irit Dinur, Shay Moran, and Amir Yehudayoff. A characterization of multiclass learnability, 2022. URL: https://arxiv.org/abs/2203.01550 .
- 7[BDPSS 09] Shai Ben-David, Dávid Pál, and Shai Shalev-Shwartz. Agnostic online learning. In COLT , 2009.
- 8[BEHW 89] Anselm Blumer, Andrzej Ehrenfeucht, David Haussler, and Manfred K Warmuth. Learnability and the Vapnik–Chervonenkis dimension. Journal of the ACM (JACM) , 36(4):929–965, 1989.