How much data is sufficient to learn high-performing algorithms? Generalization guarantees for data-driven algorithm design
Maria-Florina Balcan, Dan DeBlasio, Travis Dick, Carl Kingsford,, Tuomas Sandholm, Ellen Vitercik
TL;DR
This paper develops a general theory providing performance guarantees for data-driven algorithm tuning, applicable across various algorithms with piecewise-structured performance functions, ensuring reliable average-case performance based on training data.
Contribution
It introduces a unifying framework for deriving generalization bounds for data-driven algorithm design, covering a wide range of algorithms with complex parameter-performance relationships.
Findings
Provides bounds for algorithms with piecewise-structured performance functions
Unifies previous bounds for greedy, clustering, and integer programming algorithms
Extends guarantees to voting mechanisms and biological dynamic programming algorithms
Abstract
Algorithms often have tunable parameters that impact performance metrics such as runtime and solution quality. For many algorithms used in practice, no parameter settings admit meaningful worst-case bounds, so the parameters are made available for the user to tune. Alternatively, parameters may be tuned implicitly within the proof of a worst-case approximation ratio or runtime bound. Worst-case instances, however, may be rare or nonexistent in practice. A growing body of research has demonstrated that data-driven algorithm design can lead to significant improvements in performance. This approach uses a training set of problem instances sampled from an unknown, application-specific distribution and returns a parameter setting with strong average performance on the training set. We provide a broadly applicable theory for deriving generalization guarantees that bound the difference…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
How much data is sufficient to learn high-performing algorithms?
Generalization guarantees for data-driven algorithm design
Maria-Florina Balcan
Carnegie Mellon University
Dan DeBlasio
University of Texas at El Paso
Travis Dick
University of Pennsylvania
Carl Kingsford
Carnegie Mellon University
Ocean Genomics, Inc.
Tuomas Sandholm
Carnegie Mellon University
Optimized Markets, Inc.
Strategic Machine, Inc.
Strategy Robot, Inc.
Ellen Vitercik
Carnegie Mellon University
Abstract
Algorithms often have tunable parameters that impact performance metrics such as runtime and solution quality. For many algorithms used in practice, no parameter settings admit meaningful worst-case bounds, so the parameters are made available for the user to tune. Alternatively, parameters may be tuned implicitly within the proof of a worst-case approximation ratio or runtime bound. Worst-case instances, however, may be rare or nonexistent in practice. A growing body of research has demonstrated that data-driven algorithm design can lead to significant improvements in performance. This approach uses a training set of problem instances sampled from an unknown, application-specific distribution and returns a parameter setting with strong average performance on the training set.
We provide a broadly applicable theory for deriving generalization guarantees that bound the difference between the algorithm’s average performance over the training set and its expected performance on the unknown distribution. Our results apply no matter how the parameters are tuned, be it via an automated or manual approach. The challenge is that for many types of algorithms, performance is a volatile function of the parameters: slightly perturbing the parameters can cause a large change in behavior. Prior research [e.g., 48, 7, 10, 8] has proved generalization bounds by employing case-by-case analyses of greedy algorithms, clustering algorithms, integer programming algorithms, and selling mechanisms. We uncover a unifying structure which we use to prove extremely general guarantees, yet we recover the bounds from prior research. Our guarantees, which are tight up to logarithmic factors in the worst case, apply whenever an algorithm’s performance is a piecewise-constant, -linear, or—more generally—piecewise-structured function of its parameters. Our theory also implies novel bounds for voting mechanisms and dynamic programming algorithms from computational biology.
1 Introduction
Algorithms often have tunable parameters that impact performance metrics such as runtime, solution quality, and memory usage. These parameters may be set explicitly, as is often the case in applied disciplines. For example, integer programming solvers expose over one hundred parameters for the user to tune. There may not be parameter settings that admit meaningful worst-case bounds, but after careful parameter tuning, these algorithms can quickly find solutions to computationally challenging problems. However, applied approaches to parameter tuning have rarely come with provable guarantees. Alternatively, an algorithm’s parameters may be set implicitly, as is often the case in theoretical computer science: a proof may implicitly optimize over a parameterized family of algorithms in order to guarantee a worst-case approximation factor or runtime bound. Worst-case bounds, however, can be overly pessimistic in practice. A growing body of research (surveyed in a book chapter by Balcan [4]) has demonstrated the power of data-driven algorithm design, where machine learning is used to find parameter settings that work particularly well on problems from the application domain at hand.
We present a broadly applicable theory for proving generalization guarantees in the context of data-driven algorithm design. We adopt a natural learning-theoretic model of data-driven algorithm design introduced by Gupta and Roughgarden [48]. As in the applied literature on automated algorithm configuration [e.g., 53, 87, 106, 55, 66, 58, 107], we assume there is an unknown, application-specific distribution over the algorithm’s input instances. A learning procedure receives a training set sampled from this distribution and returns a parameter setting—or configuration—with strong average performance over the training set. If the training set is too small, this configuration may have poor expected performance. Generalization guarantees bound the difference between average performance over the training set and actual expected performance. Our guarantees apply no matter how the parameters are optimized, via an algorithmic search as in automated algorithm configuration [e.g., 87, 106, 107, 24], or manually as in experimental algorithmics [e.g., 15, 73, 57].
Across many types of algorithms—for example, combinatorial algorithms, integer programs, and dynamic programs—the algorithm’s performance is a volatile function of its parameters. This is a key challenge that distinguishes our results from prior research on generalization guarantees. For well-understood functions in machine learning theory, there is generally a simple connection between a function’s parameters and the value of the function. Meanwhile, slightly perturbing an algorithm’s parameters can cause significant changes in its behavior and performance. To provide generalization bounds, we uncover structure that governs these volatile performance functions.
The structure we discover depends on the relationship between primal and dual functions [3]. To derive generalization bounds, a common strategy is to calculate the intrinsic complexity of a function class which we refer to as the primal class. Every function is defined by a parameter setting and measures the performance of the algorithm parameterized by given the input . We measure intrinsic complexity using the classic notion of pseudo-dimension [83]. This is a challenging task because the domain of every function in is a set of problem instances, so there are no obvious notions of Lipschitz continuity or smoothness on which we can rely. Instead, we use structure exhibited by the dual class . Every dual function is defined by a problem instance and measures the algorithm’s performance as a function of its parameters given as input. The dual functions have a simple, Euclidean domain and we demonstrate that they have ample structure which we can use to bound the pseudo-dimension of .
1.1 Our contributions
Our results apply to any parameterized algorithm with dual functions that exhibit a clear-cut, ubiquitous structural property: the duals are piecewise constant, piecewise linear, or—more broadly—piecewise structured. The parameter space decomposes into a small number of regions such that within each region, the algorithm’s performance is “well behaved.”
As an example, Figure 1 illustrates a piecewise-structured function of two parameters and . There are two functions and that define a partition of the parameter space and four constant functions that define the function value on each subset from this partition.
More formally, the dual class is -piecewise decomposable if for every problem instance, there are at most boundary functions from a set (for example, the set of linear separators) that partition the parameter space into regions such that within each region, algorithmic performance is defined by a function from a set (for example, the set of constant functions). We bound the pseudo-dimension of in terms of the pseudo- and VC-dimensions of the dual classes and , denoted and . This yields our main theorem: if is the range of the functions in , then with probability over the draw of training instances, for any parameter setting, the difference between the algorithm’s average performance over the training set and its expected performance is Specifically, we prove that and that this bound is tight up to log factors. The classes and are often so well structured that bounding and is straightforward.
This is the most broadly applicable generalization bound for data-driven algorithm design in the distributional learning model that applies to arbitrary input distributions. A nascent line of research [48, 7, 10, 8, 9, 11] provides generalization bounds for a selection of parameterized algorithms. Unlike the results in this paper, those papers analyze each algorithm individually, case by case. Our approach recovers those bounds, implying guarantees for configuring greedy algorithms [48], clustering algorithms [7], and integer programming algorithms [7, 8], as well as mechanism design for revenue maximization [10]. We also derive novel generalization bounds for computational biology algorithms and voting mechanisms.
Proof insights.
At a high level, we prove this guarantee by counting the number of parameter settings with significantly different performance over any set of problem instances. To do so, we first count the number of regions induced by the boundary functions that correspond to these problem instances. This step subtly depends not on the VC-dimension of the class of boundary functions , but rather on . These boundary functions partition the parameter space into regions where across all instances in , the dual functions are simultaneously structured. Within any one region, we use the pseudo-dimension of the dual class to count the number of parameter settings in that region with significantly different performance. We aggregate these bounds over all regions to bound the pseudo-dimension of .
Parameterized dynamic programming algorithms from computational biology.
Our results imply bounds for a variety of computational biology algorithms that are used in practice. We analyze parameterized sequence alignment algorithms [45, 50, 36, 82, 81] as well as RNA folding algorithms [80], which predict how an input RNA strand would naturally fold, offering insight into the molecule’s function. We also provide guarantees for algorithms that predict topologically associating domains in DNA sequences [37], which shed light on how DNA wraps into three-dimensional structures that influence genome function.
Parameterized voting mechanisms.
A mechanism is a special type of algorithm designed to help a set of agents come to a collective decision. For example, a town’s residents may want to build a public resource such as a park, pool, or skating rink, and a mechanism would help them decide which to build. We analyze neutral affine maximizers [86, 74, 78], a well-studied family of parameterized mechanisms. The parameters can be tuned to maximize social welfare, which is the sum of the agents’ values for the mechanism’s outcome.
1.2 Additional related research
A growing body of theoretical research investigates how machine learning can be incorporated in the process of algorithm design [20, 103, 104, 61, 62, 2, 72, 85, 75, 54, 17, 48, 7, 10, 8, 9, 11, 32, 102, 31, 39, 12]. A chapter by Balcan [4] provides a survey. We highlight a few of the papers that are most related to ours below.
1.2.1 Prior research
Runtime optimization with provable guarantees.
Kleinberg et al. [61, 62] and Weisz et al. [103, 104] provide configuration procedures with provable guarantees when the goal is to minimize runtime. In contrast, our bounds apply to arbitrary performance metrics, such as solution quality as well as runtime. Also, their procedures are designed for the case where the set of parameter settings is finite (although they can still offer some guarantees when the parameter space is infinite by first sampling a finite set of parameter settings and then running the configuration procedure; Balcan et al. [8, 13] study what kinds of guarantees discretization approaches can and cannot provide). In contrast, our guarantees apply immediately to infinite parameter spaces. Finally, unlike our results, the guarantees from this prior research are not configuration-procedure-agnostic: they apply only to the specific procedures that are proposed.
Learning-augmented algorithms.
A related line of research has designed algorithms that replace some steps of a classic worst-case algorithm with a machine-learned oracle that makes predictions about structural aspects of the input [72, 85, 75, 54]. If the prediction is accurate, the algorithm’s performance (for example, its error or runtime) is superior to the original worst-case algorithm, and if the prediction is incorrect, the algorithm performs as well as that worst-case algorithm. Though similar, our approach to data-driven algorithm design is different because we are not attempting to learn structural aspects of the input; rather, we are optimizing the algorithm’s parameters directly using the training set. Moreover, we can also compete with the best-known worst-case algorithm by including it in the algorithm class over which we optimize. Just adding that one extra algorithm—however different—does not increase our sample complexity bounds. That best-in-the-worst-case algorithm does not have to be a special case of the parameterized algorithm.
Dispersion.
Prior research by Balcan, Dick, and Vitercik [9] as well as concurrent research by Balcan, Dick, and Pegden [12] provides provable guarantees for algorithm configuration, with a particular focus on online learning and privacy-preserving algorithm configuration. These tasks are impossible in the worst case, so these papers identify a property of the dual functions under which online and private configuration are possible. This condition is dispersion, which, roughly speaking, requires that the discontinuities of the dual functions are not too concentrated in any ball. Online learning guarantees imply sample complexity guarantees due to online-to-batch conversion, and Balcan et al. [9] also provide sample complexity guarantees based on dispersion using Rademacher complexity.
To prove that dispersion holds, one typically needs to show that under the distribution over problem instances, the dual functions’ discontinuities do not concentrate. This argument is typically made by assuming that the distribution is sufficiently nice or—when applicable—by appealing to the random nature of the parameterized algorithm. Thus, for arbitrary distributions and deterministic algorithms, dispersion does not necessarily hold. In contrast, our results hold even when the discontinuities concentrate, and thus are applicable to a broader set of problems in the distributional learning model. In other words, the results from this paper cannot be recovered using the techniques of Balcan et al. [9, 12].
1.2.2 Concurrent and subsequent research
Subsequently to the appearance of the original version of this paper in 2019 [16], an extensive body of research has developed that studies the use of machine learning in the context of algorithm design, as we highlight below.
Learning-augmented algorithms.
The literature on learning-augmented algorithms (summarized in the previous section) has continued to flourish in subsequent research [32, 102, 31, 27, 26, 56, 65, 65]. Some of these papers make explicit connections to the types of parameter optimization problems we study in this paper, such as research by Lavastida et al. [65], who study online flow allocation and makespan minimization problems. They formulate the machine-learned predictions as a set of parameters and study the sample complexity of learning a good parameter setting. An interesting direction for future research is to investigate which other problems from this literature can be formulated as parameter optimization algorithms, and whether the techniques in this paper can be used to derive tighter or novel guarantees.
Sample complexity bounds for data-driven algorithm design.
Chawla et al. [20] study a data-driven algorithm design problem for the Pandora’s box problem, where there is a set of alternatives with costs drawn from an unknown distribution. A search algorithm observes the alternatives’ costs one-by-one, eventually stopping and selecting one alternative. The authors show how to learn an algorithm that minimizes the selected alternative’s expected cost, plus the number of alternatives the algorithm examines. The primary contributions of that paper are 1) identifying a finite subset of algorithms that compete with the optimal algorithm, and 2) showing how to efficiently optimize over that finite subset of algorithms. Since the authors prove that they only need to optimize over a finite subset of algorithms, the sample complexity of this approach follows from a Hoeffding and union bound.
Blum et al. [17] study a data-driven approach to learning a nearly optimal cooling schedule for the simulated annealing algorithm. They provide upper and lower sample complexity bounds, with their upper bound following from a careful covering number argument. We leave as an open question whether our techniques can be combined with theirs to match their sample complexity lower bound of , where is the cooling schedule length.
Machine learning for combinatorial optimization.
A growing body of applied research has developed machine learning approaches to discrete optimization, largely with the aim of improving classic optimization algorithms such as branch-and-bound [38, 84, 95, 91, 93, 35, 108, 30, 101, 94, 63]. For example, Chmiela et al. [21] present data-driven approaches to scheduling heuristics in branch-and-bound, and they leave as an open question whether the techniques in this paper can be used to provide provable guarantees.
2 Notation and problem statement
We study algorithms parameterized by a set . As a concrete example, parameterized algorithms are often used for sequence alignment [45]. There are many features of an alignment one might wish to optimize, such as the number of matches, mismatches, or indels (defined in Section 4.1). A parameterized objective function is defined by weighting these features. As another example, hierarchical clustering algorithms often use linkage routines such as single, complete, and average linkage. Parameters can be used to interpolate between these three classic procedures [7], which can be outperformed with a careful parameter tuning [11].
We use to denote the set of problem instances the algorithm takes as input. We measure the performance of the algorithm parameterized by via a utility function , with denoting the set of all such functions. We assume there is an unknown, application-specific distribution over .
Our goal is to find a parameter vector in with high performance in expectation over the distribution . We provide generalization guarantees for this problem. Given a training set of problem instances sampled from , a generalization guarantee bounds the difference—for any choice of the parameters —between the average performance of the algorithm over and its expected performance.
Specifically, our main technical contribution is a bound on the pseudo-dimension [83] of the set . For any arbitrary set of functions that map an abstract domain to , the pseudo-dimension of , denoted , is the size of the largest set such that for some set of targets ,
[TABLE]
Classic results from learning theory [83] translate pseudo-dimension bounds into generalization guarantees. For example, suppose is the range of the functions in . For any and any distribution over , with probability over the draw of , for all functions , the difference between the average value of over and its expected value is bounded as follows:
[TABLE]
When is a set of binary-valued functions that map to , the pseudo-dimension of is more commonly referred to as the VC-dimension of [97], denoted .
3 Generalization guarantees for data-driven algorithm design
In data-driven algorithm design, there are two closely related function classes. First, for each parameter setting , measures performance as a function of the input . Similarly, for each input , there is a function defined as that measures performance as a function of the parameter vector . The set is equivalent to Assouad’s notion of the dual class [3].
Definition 3.1** (Dual class [3]).**
For any domain and set of functions , the dual class of is defined as where . Each function fixes an input and maps each function to . We refer to the class as the primal class.
The set of functions is equivalent to the dual class in the sense that for every parameter vector and every problem instance , .
Many combinatorial algorithms share a clear-cut, useful structure: for each instance , the function is piecewise structured. For example, each function might be piecewise constant with a small number of pieces. Given the equivalence of the functions and the dual class , the dual class exhibits this piecewise structure as well. We use this structure to bound the pseudo-dimension of the primal class .
Intuitively, a function is piecewise structured if we can partition the domain into subsets so that when we restrict to one piece , equals some well-structured function . In other words, for all , . We define the partition using boundary functions . Each function divides the domain into two sets: the points it labels 0 and the points it labels 1.
Figure 2 illustrates a partition of by boundary functions. Together, the boundary functions partition the domain into at most regions, each one corresponding to a bit vector that describes on which side of each boundary the region belongs. For each region, we specify a piece function that defines the function values of restricted to that region. Figure 1 shows an example of a piecewise-structured function with two boundary functions and four piece functions.
For many parameterized algorithms, every function in the dual class is piecewise structured. Moreover, across dual functions, the boundary functions come from a single, fixed class, as do the piece functions. For example, the boundary functions might always be halfspace indicator functions, while the piece functions might always be linear functions. The following definition captures this structure.
Definition 3.2**.**
A function class that maps a domain to is -piecewise decomposable for a class of boundary functions and a class of piece functions if the following holds: for every , there are boundary functions and a piece function for each bit vector such that for all , where
Our main theorem shows that whenever the dual class is -piecewise decomposable, we can bound the pseudo-dimension of in terms of the VC-dimension of and the pseudo-dimension of . Later, we show that for many common classes and , we can easily bound the complexity of their duals.
Theorem 3.3**.**
Suppose that the dual function class is -piecewise decomposable with boundary functions and piece functions . The pseudo-dimension of is bounded as follows:
[TABLE]
To help make the proof of Theorem 3.3 succinct, we extract a key insight in the following lemma. Given a set of functions that map a domain to , Lemma 3.4 bounds the number of binary vectors
[TABLE]
we can obtain for any functions as we vary the input . Pictorially, if we partition using the functions , , and from Figure 2 for example, Lemma 3.4 bounds the number of regions in the partition. This bound depends not on the VC-dimension of the class , but rather on that of its dual . We use a classic lemma by Sauer [90] to prove Lemma 3.4. Sauer’s lemma [90] bounds the number of binary vectors of the form we can obtain for any elements as we vary the function by . Therefore, it does not immediately imply a bound on the number of vectors from Equation (3). In order to apply Sauer’s lemma, we must transition to the dual class.
Lemma 3.4**.**
Let be a set of functions that map a domain to . For any functions , the number of binary vectors obtained by varying the input is bounded as follows:
[TABLE]
Proof.
We rewrite the left-hand-side of Equation (4) as Since we fix inputs and vary the function , the lemma statement follows from Sauer’s lemma [90]. ∎
We now prove Theorem 3.3.
Proof of Theorem 3.3.
Fix an arbitrary set of problem instances and targets . We bound the number of ways that can label the problem instances with respect to the target thresholds . In other words, as per Equation (1), we bound the size of the set
[TABLE]
by . Then solving for the largest such that
[TABLE]
gives a bound on the pseudo-dimension of . Our bound on Equation (5) has two main steps:
In Claim 3.5, we show that there are subsets partitioning the parameter space such that within any one subset, the dual functions are simultaneously structured. In particular, for each subset , there exist piece functions such that for all parameter settings and . This is the partition of induced by aggregating all of the boundary functions corresponding to the dual functions . 2. 2.
We then show that for any region of the partition, as we vary the parameter vector , can label the problem instances in at most ways with respect to the target thresholds . It follows that the total number of ways that can label the problem instances is bounded by .
We now prove the first claim.
Claim 3.5**.**
There are subsets partitioning the parameter space such that within any one subset, the dual functions are simultaneously structured. In particular, for each subset , there exist piece functions such that for all parameter settings and .
Proof of Claim 3.5.
Let be the dual functions corresponding to the problem instances . Since is -piecewise decomposable, we know that for each function , there are boundary functions that define its piecewise decomposition. Let be the union of these boundary functions across all . For ease of notation, we relabel the functions in , calling them . Let be the total number of -dimensional vectors we can obtain by applying the functions in to elements of :
[TABLE]
By Lemma 3.4, Let be the binary vectors in the set from Equation (6). For each , let By construction, for each set , the values of all the boundary functions are constant as we vary . Therefore, there is a fixed set of piece functions so that for all parameter vectors and indices . Therefore, the claim holds. ∎
Claim 3.5 and Equation (5) imply that for every subset of the partition,
[TABLE]
Lemma 3.4 implies that Equation (7) is bounded by . In other words, for any region of the partition, as we vary the parameter vector , can label the problem instances in at most ways with respect to the target thresholds . Because there are regions of the partition, we can conclude that can label the problem instances in at most distinct ways relative to the targets . In other words, Equation (5) is bounded by . On the other hand, if shatters the problem instances , then the number of distinct labelings must be . Therefore, the pseduo-dimension of is at most the largest value of such that , which implies that
[TABLE]
as claimed. ∎
We prove several lower bounds which show that Theorem 3.3 is tight up to logarithmic factors.
Theorem 3.6**.**
The following lower bounds hold:
There is a parameterized sequence alignment algorithm with for some . Its dual class is -piecewise decomposable for classes and with . 2. 2.
There is a parameterized voting mechanism with for some . Its dual class is -piecewise decomposable for classes and with and .
Proof.
In Theorem 4.3, we prove the result for sequence alignment, in which case is the maximum length of the sequences, is the set of constant functions, and is the set of threshold functions. In Theorem 5.2, we prove the result for voting mechanisms, in which case is the number of agents that participate in the mechanism, is the set of constant functions, and is the set of homogeneous linear separators in . ∎
Applications of our main theorem to representative function classes
We now instantiate Theorem 3.3 in a general setting inspired by data-driven algorithm design.
One-dimensional functions with a bounded number of oscillations.
Let be a set of utility functions defined over a single-dimensional parameter space. We often find that the dual functions are piecewise constant, linear, or polynomial. More generally, the dual functions are piecewise structured with piece functions that oscillate a fixed number of times. In other words, the dual class is -piecewise decomposable where the boundary functions are thresholds and the piece functions oscillate a bounded number of times, as formalized below.
Definition 3.7**.**
A function has at most oscillations if for every , the function is piecewise constant with at most discontinuities.
Figure 3 illustrates three common types of functions with bounded oscillations. In the following lemma, we prove that if is a class of functions that map to , each of which has at most oscillations, then .
Lemma 3.8**.**
Let be a class of functions mapping to , each of which has at most oscillations. Then .
Proof.
Suppose that . Then there exist functions and witnesses such that for every subset , there exists a parameter setting such that if and only if . We can simplify notation as follows: since for every function , we have that for every subset , there exists a parameter setting such that if and only if . Let be the set of parameter settings corresponding to each subset . By definition, these parameter settings induce distinct binary vectors as follows:
[TABLE]
On the other hand, since each function has at most oscillations, we can partition into intervals such that for every interval and every , the function is constant across the interval . Therefore, at most one parameter setting can fall within a single interval . Otherwise, if , then
[TABLE]
which is a contradiction. As a result, . The lemma then follows from Lemma A.1. ∎
Lemma 3.8 implies the following pseudo-dimension bound when the dual function class is -piecewise decomposable, where the boundary functions are thresholds and the piece functions oscillate a bounded number of times.
Lemma 3.9**.**
Let be a set of utility functions and suppose the dual class is -decomposable, where the boundary functions are thresholds . Suppose for each , the function has at most oscillations. Then .
Proof.
First, we claim that . For a contradiction, suppose can shatter two functions , where . There must be a parameter setting where and . Therefore, , which is a contradiction, so .
Next, we claim that . For each function , let be defined as . By assumption, each function has at most oscillations. Let and let . By Lemma 3.8, we know that . We claim that . For a contradiction, suppose the class can shatter functions using witnesses . By definition, this means that
[TABLE]
For any function and any parameter setting , . Therefore,
[TABLE]
which contradicts the fact that . Therefore, . The corollary then follows from Theorem 3.3. ∎
Multi-dimensional piecewise-linear functions.
In many data-driven algorithm design problems, we find that the boundary functions correspond to halfspace thresholds and the piece functions correspond to constant or linear functions. We handle this case in the following lemma.
Lemma 3.10**.**
Let be a class of utility functions defined over a -dimensional parameter space. Suppose the dual class is -piecewise decomposable, where the boundary functions are halfspace indicator functions and the piece functions are linear functions . Then .
The proof of this lemma follows from classic VC- and pseudo-dimension bounds for linear functions and can be found in Appendix B.
4 Parameterized computational biology algorithms
We study algorithms that are used in practice for three biological problems: sequence alignment, RNA folding, and predicting topologically associated domains in DNA. In these applications, there are two unifying similarities. First, algorithmic performance is measured in terms of the distance between the algorithm’s output and a ground-truth solution. In most cases, this solution is discovered using laboratory experimentation, so it is only available for the instances in the training set. Second, these algorithms use dynamic programming to maximize parameterized objective functions. This objective function represents a surrogate optimization criterion for the dynamic programming algorithm, whereas utility measures how well the algorithm’s output resembles the ground truth. There may be multiple solutions that maximize this objective function, which we call co-optimal. Although co-optimal solutions have the same objective function value, they may have different utilities. To handle tie-breaking, we assume that in any region of the parameter space where the set of co-optimal solutions is fixed, the algorithm’s output is also fixed, which is typically true in practice.
4.1 Sequence alignment
4.1.1 Global pairwise sequence alignment
In pairwise sequence alignment, the goal is to line up strings in order to identify regions of similarity. In biology, for example, these similar regions indicate functional, structural, or evolutionary relationships between the sequences. Formally, let be an alphabet and let be two sequences. A sequence alignment is a pair of sequences such that , , and where del is a function that deletes every , or gap character. There are many features of an alignment that one might wish to optimize, such as the number of matches (), mismatches (), indels ( or ), and gaps (maximal sequences of consecutive gap characters in ). We denote these features using functions that map pairs of sequences and alignments to .
A common dynamic programming algorithm [45, 100] returns the alignment that maximizes the objective function
[TABLE]
where is a parameter vector. We denote the output alignment as . As Gusfield, Balasubramanian, and Naor [50] wrote, “there is considerable disagreement among molecular biologists about the correct choice” of a parameter setting .
We assume that there is a utility function that characterizes an alignment’s quality, denoted . For example, might measure the distance between and a “ground truth” alignment of and [89]. We then define to be the utility of the alignment returned by the algorithm .
In the following lemma, we prove that the set of utility functions has piecewise-structured dual functions.
Lemma 4.1**.**
Let be the set of functions that map sequence pairs to . The dual class is -piecewise decomposable, where consists of constant functions and consists of halfspace indicator functions .
Proof.
Fix a sequence pair and . Let be the set of alignments the algorithm returns as we range over all parameter vectors . In other words, . In Lemma C.1, we prove that . For any alignment , the algorithm will return if and only if
[TABLE]
for all . Therefore, there is a set of at most hyperplanes such that across all parameter vectors in a single connected component of , the output of the algorithm parameterized by , , is fixed. (As is standard, indicates set removal.) This means that for any connected component of , there exists a real value such that for all . By definition of the dual, this means that as well.
We now use this structure to show that the dual class is -piecewise decomposable, as per Definition 3.2. Recall that consists of halfspace indicator functions and consists of constant functions . For each pair of alignments , let correspond to the halfspace represented in Equation (9). Order these functions arbitrarily as . Every connected component of corresponds to a sign pattern of the hyperplanes. For a given region , let be the corresponding sign pattern. Define the function as , so for all . Meanwhile, for every vector not corresponding to a sign pattern of the hyperplanes, let , so for all . In this way, for every ,
[TABLE]
as desired. ∎
Lemmas 3.10 and 4.1 imply that In Appendix C, we also prove tighter guarantees for a structured subclass of algorithms [45, 100]. In that case, and is the number of matches in the alignment, is the number of mismatches, is the number of indels, and is the number of gaps. Building on prior research [50, 36, 81], we show (Lemma 4.2) that the dual class is -piecewise decomposable with and defined as in Lemma 4.1. This implies a pseudo-dimension bound of , which is significantly tighter than that of Lemma 4.1. We also prove that this pseudo-dimension bound is tight with a lower bound of (Theorem 4.3). Moreover, in Appendix 4.1.3, we provide guarantees for algorithms that align more than two sequences.
4.1.2 Tighter guarantees for a structured algorithm subclass: the affine-gap model
A line of prior work [50, 36, 82, 81] analyzed a specific instantiation of the objective function (8) where . In this case, we can obtain a pseudo-dimension bound of , which is exponentially better than the bound implied by Lemma 4.1. Given a pair of sequences , the dynamic programming algorithm returns the alignment maximizes the objective function
[TABLE]
where equals the number of matches, is the number of mismatches, is the number of indels, is the number of gaps, and is a parameter vector. We denote the output alignment as . This is known as the affine-gap scoring model. We exploit specific structure exhibited by this algorithm family to obtain the exponential pseudo-dimension improvement. This useful structure guarantees that for any pair of sequences and , there are only different alignments the algorithm family might produce as we range over parameter vectors [50, 36, 81]. This bound is exponentially smaller than our generic bound of from Lemma C.1.
Lemma 4.2**.**
Let be the set of functions
[TABLE]
that map sequence pairs to . The dual class is -piecewise decomposable, where consists of constant functions and where consists of halfspace indicator functions .
Proof.
Fix a sequence pair and . Let be the set of alignments the algorithm returns as we range over all parameter vectors . In other words, . From prior research [50, 36, 81], we know that . For any alignment , the algorithm will return if and only if
[TABLE]
for all . Therefore, there is a set of at most hyperplanes such that across all parameter vectors in a single connected component of , the output of the algorithm parameterized by , , is fixed. The proof now follows by the exact same logic as that of Lemma 4.1. ∎
Lemmas 3.10 and 4.2 imply that We also prove that this pseudo-dimension bound is tight up to constant factors. In this lower bound proof, our utility function is the Q score between a given alignment of two sequences and the ground-truth alignment (the Q score is also known as the SPS score in the case of multiple sequence alignment [28]). The score between and the ground-truth alignment is the fraction of aligned letter pairs in that are correctly reproduced in . For example, the following alignment has a Q score of because it correctly aligns the two pairs of Cs, but not the pair of Gs:
[TABLE]
We use the notation to denote the Q score between and the ground-truth alignment of and . The full proof of the following theorem is in Appendix C.
Theorem 4.3**.**
There exists a set of co-optimal-constant algorithms and an alphabet such that the set of functions which map sequence pairs of length at most to , has a pseudo-dimension of .
Proof sketch.
In this proof sketch, we illustrate the way in which two sequences pairs can be shattered, and then describe how the proof can be generalized to sequence pairs.
Setup.
Our setup consists of the following three elements: the alphabet, the two sequence pairs and , and ground-truth alignments of these pairs. We detail these elements below:
Our alphabet consists of twelve characters: . 2. 2.
The two sequence pairs are comprised of three subsequence pairs: , , and , where
[TABLE]
We define the two sequence pairs as
[TABLE] 3. 3.
Finally, we define ground-truth alignments of the sequence pairs and . We define the ground-truth alignment of to be
[TABLE]
The most important properties of this alignment are that the characters are always matching and the characters alternate between matching and not matching. Similarly, we define the ground-truth alignment of the pair to be
[TABLE]
Shattering.
We now show that these two sequence pairs can be shattered. A key step is proving that the functions and have the following form:
[TABLE]
The form of is illustrated by Figure 4. It is then straightforward to verify that the two sequence pairs are shattered by the parameter settings , , , and with the witnesses . In other words, the mismatch and gap parameters are set to [math] and the indel parameter takes the values .
Proof sketch of Equation (12).
The full proof that Equation (12) holds follows the following high-level reasoning:
First, we prove that under the algorithm’s output alignment, the characters will always be matching. Intuitively, this is because the algorithm’s objective function will always be maximized when each subsequence is aligned with . 2. 2.
Second, we prove that the characters will be matched if and only if . Intuitively, this is because in order to match these characters, we must pay with indels. Since the objective function is , the match will be worth the indels if and only if .
These two properties in conjunction mean that when , none of the characters are matched, so the characters that are correctly aligned (as per the ground-truth alignment (Equation (11))) in the algorithm’s output are , , , , and , as illustrated by purple in the top alignment of Figure 4. Since there are a total of aligned letters in the ground-truth alignment, we have that the Q score is , or in other words,
When shifts to the next-smallest interval , the indel penalty is sufficiently small that the characters will align. Thus we lose the correct alignment , and the Q score drops to . Similarly, if we decrease to the next-smallest interval , the characters will align, which is correct under the ground-truth alignment (Equation (11)). Thus the Q score increases back to . Finally, by the same logic, when , we lose the correct alignment in favor of the alignment of the characters, so the Q score falls to . In this way, we prove the form of from Equation (12). A parallel argument proves the form of .
Generalization to shattering sequence pairs.
This proof intuition naturally generalizes to sequence pairs of length by expanding the number of subsequences a la Equation (10). In essence, if we define and for a carefully-chosen , then we can force to oscillate times. Similarly, if we define and , then we can force to oscillate half as many times, and so on. This construction allows us to shatter sequences. ∎
4.1.3 Progressive multiple sequence alignment
The multiple sequence alignment problem is a generalization of the pairwise alignment problem introduced in Section 4.1. Let be an abstract alphabet and let be a collection of sequences in of length . A multiple sequence alignment is a collection of sequences such that the following hold:
The aligned sequences are the same length: . 2. 2.
Removing the gap characters from gives : for all , . 3. 3.
For every position in the alignment, at least one of the aligned sequences has a non-gap character. In other words, for every position , there exists a sequence such that .
The extension from pairwise to multiple sequence alignment is computationally challenging: all common formulations of the problem are NP-complete [99, 59]. As a result, heuristics have been developed to find good but possibly sub-optimal alignments. The most common heuristic approach is called progressive multiple sequence alignment. It leverages efficient pairwise alignment algorithms to heuristically align multiple sequences [34].
The input to a progressive multiple sequence alignment algorithm is a collection of sequences together with a binary guide tree with leaves111The problem of constructing the guide tree is also an algorithmic task, often tackled via hierarchical clustering, but we are agnostic to that pre-processing step.. The tree indicates how the original alignment should be decomposed into a hierarchy of subproblems, each of which can be heuristically solved using pairwise alignment. The leaves of the guide tree correspond to the input sequences .
While there are many formalizations of the progressive alignment method, for the sake of analysis we will focus on “partial consensus” described by Higgins and Sharp [51]. Here, we provide a high-level description of the algorithm and in Appendix C.2, we include more detailed pseudo-code. At a high level, the algorithm recursively constructs an alignment in two stages: first, it creates a consensus sequence for each node in the guide tree using a pairwise alignment algorithm, and then it propagates the node-level alignments to the leaves by inserting additional gap characters.
In a bit more detail, for each node in the tree, we construct an alignment of the consensus sequences of its children as well as a consensus sequence . Since each leaf corresponds to a single input sequence, it has a trivial alignment and the consensus sequence is just the input sequence itself. For an internal node with children and , we use a pairwise alignment algorithm to construct an alignment of the consensus strings and . Finally, we define the consensus sequence of the node to be the string such that is the most-frequent non-gap character in the position in the alignment . By defining the consensus sequence in this way, we can represent all of the sub-alignments of the leaves of the subtree rooted at as a single sequence which can be aligned using existing methods. We obtain a full multiple sequence alignment by iteratively replacing each consensus sequence by the pairwise alignment it represents, adding gap columns to the sub-alignments when necessary. Once we add a gap to a sequence, we never remove it: “once a gap, always a gap.”
The family of parameterized pairwise alignment algorithms introduced in Section 4.1 induces a parameterized family of progressive multiple sequence alignment algorithms . In particular, the algorithm takes as input a collection of input sequences and a guide tree , and it outputs a multiple-sequence alignment by applying the pairwise alignment algorithm at each node of the guide tree. We assume that there is a utility function that characterizes an alignment’s quality, denoted . We then define to be the utility of the alignment returned by the algorithm . The proof of the following lemma is in Appendix C.2. It follows by the same logic as Lemma 4.1 for pairwise sequence alignment, inductively over the guide tree.
Lemma 4.4**.**
Let be a guide tree of depth and let be the set of functions
[TABLE]
The dual class is
[TABLE]
where consists of halfspace indicator functions and consists of constant functions .
This lemma together with Lemma 3.10 implies that . Therefore, the pseudo-dimension grows only linearly in and quadratically in in the affine-gap model () when the guide tree is balanced ().
4.2 RNA folding
RNA molecules have many essential roles including protein coding and enzymatic functions [52]. RNA is assembled as a chain of bases denoted A, U, C, and G. It is often found as a single strand folded onto itself with non-adjacent bases physically bound together. RNA folding algorithms infer the way strands would naturally fold, shedding light on their functions. Given a sequence , we represent a folding by a set of pairs . If , then the and bases of bind together. Typically, the bases A and U bind together, as do C and G. Other matchings are likely less stable. We assume that the foldings do not contain any pseudoknots, which are pairs that cross with .
A well-studied algorithm returns a folding that maximizes a parameterized objective function [80]. At a high level, this objective function trades off between global properties of the folding (the number of binding pairs ) and local properties (the likelihood that bases would appear close together in the folding). Specifically, the algorithm uses dynamic programming to return the folding that maximizes
[TABLE]
where is a parameter and is a score for having neighboring pairs of the letters and . These scores help identify stable sub-structures.
We assume there is a utility function that characterizes a folding’s quality, denoted . For example, might measure the fraction of pairs shared between and a “ground-truth” folding, obtained via expensive computation or laboratory experiments.
Lemma 4.5**.**
Let be the set of functions . The dual class is -piecewise decomposable, where consists of threshold functions and consists of constant functions .
Proof.
Fix a sequence . Let be the set of alignments that the algorithm returns as we range over all parameters . In other words, . We know that every folding has length at most . For any , let be the folding of length that maximizes the right-hand-side of Equation (13):
[TABLE]
The folding the algorithm returns will always be one of , so
Fix an arbitrary folding . We know that will be the folding returned by the algorithm if and only if
[TABLE]
for all . Since these functions are linear in , this means there is a set of intervals with such that for any one interval , across all , is fixed. This means that for any one interval , there exists a real value such that for all . By definition of the dual, this means that as well.
We now use this structure to show that the dual class is -piecewise decomposable, as per Definition 3.2. Recall that consists of threshold functions and consists of constant functions . We claim that there exists a function for every vector such that for every ,
[TABLE]
To see why, suppose for some . Then for all and for all . Let be the vector that has only 0’s in its first coordinates and all ’s in its remaining coordinates. For all , we define , so for all . For any other , we set , so for all . Therefore, Equation (15) holds. ∎
Since constant functions have zero oscillations, Lemmas 3.9 and 4.5 imply that
4.3 Prediction of topologically associating domains
Inside a cell, the linear DNA of the genome wraps into three-dimensional structures that influence genome function. Some regions of the genome are closer than others and thereby interact more. Topologically associating domains (TADs) are contiguous segments of the genome that fold into compact regions. More formally, given the genome length , a TAD set is a set such that . If , the bases within the corresponding substring physically interact more frequently with each other than with other bases. Disrupting TAD boundaries can affect the expression of nearby genes, which can trigger diseases such as congenital malformations and cancer [71].
The contact frequency of any two genome locations, denoted by a matrix , can be measured via experiments [67]. A dynamic programming algorithm introduced by Filippova et al. [37] returns the TAD set that maximizes
[TABLE]
where is a parameter,
[TABLE]
is the scaled density of the subgraph induced by the interactions between genomic loci and , and
[TABLE]
is the mean value of over all sub-matrices of length along the diagonal of . We note that unlike the sequence alignment and RNA folding algorithms, the parameter appears in the exponent of the objective function.
We assume there is a utility function that characterizes the quality of a TAD set , denoted . For example, might measure the fraction of TADs in that are in the correct location with respect to a ground-truth TAD set.
Lemma 4.6**.**
Let be the set of functions . The dual class is -piecewise decomposable, where consists of threshold functions and consists of constant functions .
Proof.
Fix a matrix . We begin by rewriting Equation (16) as follows:
[TABLE]
where
[TABLE]
is a constant that does not depend on .
Let be the set of TAD sets that the algorithm returns as we range over all parameters . In other words, . Since each TAD set is a subset of , . For any TAD set , the algorithm will return if and only if
[TABLE]
for all . This means that as we range over the positive reals, the TAD set returned by algorithm will only change when
[TABLE]
for some . As a result of Rolle’s Theorem (Corollary A.3), we know that Equation (17) has at most solutions. This means there are intervals with that partition such that across all within any one interval , the TAD set returned by algorithm is fixed. Therefore, there exists a real value such that for all . By definition of the dual, this means that as well.
We now use this structure to show that the dual class is -piecewise decomposable, as per Definition 3.2. Recall that consists of threshold functions and consists of constant functions . We claim that there exists a function for every vector such that for every ,
[TABLE]
To see why, suppose for some . Then for all and for all . Let be the vector that has only 0’s in its first coordinates and all ’s in its remaining coordinates. For all , we define , so for all . For any other , we set , so for all . Therefore, Equation (18) holds. ∎
Since constant functions have zero oscillations, Lemmas 3.9 and 4.6 imply that
5 Parameterized voting mechanisms
A large body of research in economics studies how to design protocols—or mechanisms—that help groups of agents come to collective decisions. For example, when children inherit an estate, how should they divide the property? When a jointly-owned company is dissolved, which partner should buy the others out? There is no one protocol that best answers these questions; the optimal mechanism depends on the setting at hand.
We study a family of mechanisms called neutral affine maximizers (NAMs) [86, 74, 78]. A NAM takes as input a set of agents’ reported values for each possible outcome and returns one of those outcomes. A NAM can thus be thought of as an algorithm that the agents use to arrive at a single outcome. NAMs are incentive compatible, which means that each agent is incentivized to report his values truthfully. In order to satisfy incentive compatibility, each agent may have to make a payment. NAMs are also budget-balanced which means that the aggregated payments are redistributed among the agents.
Formally, we study a setting where there is a set of alternatives and a set of agents. Each agent has a value for each alternative . We denote all of his values as and all agents’ values as . In this case, the unknown distribution is over vectors .
A NAM is defined by parameters (one per agent) such that at least one agent is assigned a weight of zero. There is a social choice function which uses the values to choose an alternative . In particular, maximizes the agents’ weighted values. Each agent with zero weight is called a sink agent because his values do not influence the outcome. For every agent who is not a sink agent , their payment is defined as in the weighted version of the classic Vickrey-Clarke-Groves mechanism [98, 22, 46]. To achieve budget balance, these payments are given to the sink agent(s). More formally, let and for each agent , let The payment function is defined as
[TABLE]
We aim to optimize the expected social welfare of the NAM’s outcome , so we define the utility function .
Lemma 5.1**.**
Let be the set of functions . The dual class is -piecewise decomposable, where consists of halfspace indicators and consists of constant functions .
Proof.
Fix a valuation vector . We know that for any two alternatives , the alternative would be selected over so long as
[TABLE]
Therefore, there is a set of hyperplanes such that across all parameter vectors in a single connected component of , the outcome of the NAM defined by is fixed. When the outcome of the NAM is fixed, the social welfare is fixed as well. This means that for a single connected component of , there exists a real value such that for all . By definition of the dual, this means that as well.
We now use this structure to show that the dual class is -piecewise decomposable, as per Definition 3.2. Recall that consists of halfspace indicator functions and consists of constant functions . For each pair of alternatives , let correspond to the halfspace represented in Equation (19). Order these functions arbitrarily as . Every connected component of corresponds to a sign pattern of the hyperplanes. For a given region , let be the corresponding sign pattern. Define the function as , so for all . Meanwhile, for every vector not corresponding to a sign pattern of the hyperplanes, let , so for all . In this way, for every ,
[TABLE]
as desired. ∎
Theorem 3.3 and Lemma 5.1 imply that the pseudo-dimension of is Next, we prove that the pseudo-dimension of is at least , which means that our pseudo-dimension upper bound is tight up to log factors.
Theorem 5.2**.**
Let be the set of functions . Then .
Proof.
Let the number of alternatives and without loss of generality, suppose that is even. To prove this theorem, we will identify a set of valuation vectors that are shattered by the set of social welfare functions.
Let be an arbitrary number in . For each , define agent ’s values for the first and second alternatives under the valuation vector —namely, and —as follows:
[TABLE]
For example, if there are agents, then across the valuation vectors , the agents’ values for the first alternative are defined as
[TABLE]
and their values for the second alternative are defined as
[TABLE]
Let be an arbitrary bit vector. We will construct a NAM parameter vector such that for any , if , then the outcome of the NAM given bids will be the second alternative, so because there is always exactly one agent who has a value of for the second alternative, and every other agent has a value of [math]. Meanwhile, if , then the outcome of the NAM given bids will be the first alternative, so because there is always exactly one agent who has a value of for the first alternative, and every other agent has a value of [math]. To do so, when , must ignore the values of agent in favor of the values of agent . After all, under , agent has a value of for the first alternative and agent has a value of for the second alternative, and all other values are [math]. By a similar argument, when , must ignore the values of agent in favor of the values of agent . Specifically, we define as follows: for all , if , then and and if , then and . All other entries of are set to [math].
We claim that if , then . To see why, we know that . Meanwhile, . Therefore, the outcome of the NAM is alternative 2. The social welfare of this alternative is , so .
Next, we claim that if , then . To see why, we know that . Meanwhile, . Therefore, the outcome of the NAM is alternative 1. The social welfare of this alternative is , so .
We conclude that the valuation vectors that are shattered by the set of social welfare functions with witnesses . ∎
Theorem 5.2 implies that the pseudo-dimension upper bound from Lemma 5.1 is tight up to logarithmic factors.
6 Subsumption of prior research on generalization guarantees
Theorem 3.3 also recovers existing guarantees for data-driven algorithm design. In all of these cases, Theorem 3.3 implies generalization guarantees that match the existing bounds, but in many cases, our approach provides a more succinct proof.
In Section 6.1, we analyze several parameterized clustering algorithms [7], which have piecewise-constant dual functions. These algorithms first run a linkage routine which builds a hierarchical tree of clusters. The parameters interpolate between the popular single, average, and complete linkage. The linkage routine is followed by a dynamic programming procedure that returns a clustering corresponding to a pruning of the hierarchical tree. 2. 2.
Balcan et al. [11] study a family of linkage-based clustering algorithms where the parameters control the distance metric used for clustering in addition to the linkage routine. The algorithm family has two sets of parameters. The first set of parameters interpolate between linkage algorithms, while the second set interpolate between distance metrics. The dual functions are piecewise-constant with quadratic boundary functions. We recover their generalization bounds in Section 6.1.2. 3. 3.
In Section 6.2, we analyze several integer programming algorithms, which have piecewise-constant and piecewise-inverse-quadratic dual functions (as in Figure 3(c)). The first is branch-and-bound, which is used by commercial solvers such as CPLEX. Branch-and-bound always finds an optimal solution and its parameters control runtime and memory usage. We also study semidefinite programming approximation algorithms for integer quadratic programming. We analyze a parameterized algorithm introduced by Feige and Langberg [33] which includes the Goemans-Williamson algorithm [41] as a special case. We recover previous generalization bounds in both settings [8, 7]. 4. 4.
Gupta and Roughgarden [48] introduced parameterized greedy algorithms for the knapsack and maximum weight independent set problems, which we show have piecewise-constant dual functions. We recover their generalization bounds in Section 6.3. 5. 5.
We provide generalization bounds for parameterized selling mechanisms when the goal is to maximize revenue, which have piecewise-linear dual functions (as in Figure 3(b)). A long line of research has studied revenue maximization via machine learning [68, 69, 88, 5, 29, 23, 25, 43, 47, 19, 44, 77, 76]. In Section 6.4, we recover Balcan, Sandholm, and Vitercik’s generalization bounds [10] which apply to a variety of pricing, auction, and lottery mechanisms. They proved new bounds for mechanism classes not previously studied in the sample-based mechanism design literature and matched or improved over the best known guarantees for many classes.
6.1 Clustering algorithms
A clustering instance is made up of a set points from a data domain and a distance metric . The goal is to split up the points into groups, or “clusters,” so that within each group, distances are minimized and between each group, distances are maximized. Typically, a clustering’s quality is quantified by some objective function. Classic choices include the -means, -median, or -center objective functions. Unfortunately, finding the clustering that minimizes any one of these objectives is NP-hard. Clustering algorithms have uses in data science, computational biology [79], and many other fields.
Balcan et al. [7, 11] analyze agglomerative clustering algorithms. This type of algorithm requires a merge function , defining the distances between point sets . The algorithm constructs a cluster tree. This tree starts with leaf nodes, each containing a point from . Over a series of rounds, the algorithm merges the sets with minimum distance according to . The tree is complete when there is one node remaining, which consists of the set . The children of each internal node consist of the two sets merged to create the node. There are several common merge function : (single-linkage), (average-linkage), and (complete-linkage). Following the linkage procedure, there is a dynamic programming step. This steps finds the tree pruning that minimizes an objective function, such as the -means, -median, or -center objectives.
To evaluate the quality of a clustering, we assume access to a utility function where is the set of all cluster trees over the data domain . For example, might measure the distance between the ground truth clustering and the optimal -means pruning of the cluster tree .
In Section 6.1.1, we present results for learning merge functions and in Section 6.1.2, we present results for learning distance functions in addition to merge functions. The latter set of results apply to a special subclass of merge functions called two-point-based (as we describe in Section 6.1.2), and thus do not subsume the results in Section 6.1.1, but do apply to the more general problem of learning a distance function in addition to a merge function.
6.1.1 Learning merge functions
Balcan, Nagarajan, Vitercik, and White [7] study several families of merge functions:
[TABLE]
The classes and interpolate between single- ( and ) and complete-linkage ( and ). The class includes as special cases average-, complete-, and single-linkage.
For each class and each parameter , let be the algorithm that takes as input a clustering instance and returns a cluster tree .
Balcan et al. [7] prove the following useful structure about the classes and :
Lemma 6.1** ([7]).**
Let be an arbitrary clustering instance over points. There is a partition of into intervals such that for any interval and any two parameters , the sequences of merges the agglomerative clustering algorithm makes using the merge functions and are identical. The same holds for the set of merge functions .
This structure immediately implies that the corresponding class of utility functions has a piecewise-structured dual class.
Corollary 6.2**.**
Let be the set of functions
[TABLE]
mapping clustering instances to . The dual class is -piecewise decomposable, where consists of threshold functions and consists of constant functions . The same holds when is defined according to merge functions in as
Lemma 3.9 and Corollary 6.2 imply the following pseudo-dimension bound.
Corollary 6.3**.**
Let be the set of functions
[TABLE]
mapping clustering instances to . Then . The same holds when is defined according to merge functions in as
Balcan et al. [7] prove a similar guarantee for the more complicated class .
Lemma 6.4** ([7]).**
Let be an arbitrary clustering instance over points. There is a partition of into intervals such that for any interval and any two parameters , the sequences of merges the agglomerative clustering algorithm makes using the merge functions and are identical.
Again, this structure immediately implies that the corresponding class of utility functions has a piecewise-structured dual class.
Corollary 6.5**.**
Let be the set of functions
[TABLE]
mapping clustering instances to . The dual class is -piecewise decomposable, where consists of threshold functions and consists of constant functions .
Lemma 3.9 and Corollary 6.5 imply the following pseudo-dimension bound.
Corollary 6.6**.**
Let be the set of functions
[TABLE]
mapping clustering instances to . Then .
Corollaries 6.3 and 6.6 match the pseudo-dimension guarantees that Balcan et al. [7] prove.
6.1.2 Learning merge functions and distance functions
Balcan, Dick, and Lang [11] extend the clustering generalization bounds of Balcan, Nagarajan, Vitercik, and White [7] to the case of learning both a distance metric and a merge function. They introduce a family of linkage-based clustering algorithms that simultaneously interpolate between a collection of base metrics and base merge functions . The algorithm family is parameterized by , where and are mixing weights for the merge functions and metrics, respectively. The algorithm with parameters starts with each point in a cluster of its own and repeatedly merges the pair of clusters and minimizing , where
[TABLE]
We use the notation to denote the algorithm that takes as input a clustering instance and returns a cluster tree using the merge function , where .
When analyzing this algorithm family, Balcan et al. [11] prove that the following piecewise-structure holds when all of the merge functions are two-point-based, which roughly requires that for any pair of clusters and , there exist points and such that . Single- and complete-linkage are two-point-based, but average-linkage is not.
Lemma 6.7** ([11]).**
For any clustering instance , there exists a collection of quadratic boundary functions that partition the -dimensional parameter space into regions where the algorithm’s output is constant on each region in the partition.
This lemma immediately implies that the corresponding class of utility functions has a piecewise-structured dual class.
Corollary 6.8**.**
Let be the set of functions mapping clustering instances to . The dual class is -piecewise decomposable, where is the set of constant functions and is the set of quadratic functions defined on .
Using the fact that , we obtain the following pseudo-dimension bound.
Corollary 6.9**.**
Let be the set of functions mapping clustering instances to . Then
[TABLE]
This matches the generalization bound that Balcan et al. [11] prove.
6.2 Integer programming
Several papers [7, 8] study data-driven algorithm design for both integer linear and integer quadratic programming, as we describe below.
Integer linear programming.
In the context of integer linear programming, Balcan et al. [8] focus on branch-and-bound (B&B) [64], an algorithm for solving mixed integer linear programs (MILPs). A MILP is defined by a matrix , a vector , a vector , and a set of indices . The goal is to find a vector such that is maximized, , and for every index , is constrained to be binary: .
Branch-and-bound builds a search tree to solve an input MILP . At the root of the search tree is the original MILP . At each round, the algorithm chooses a leaf of the search tree, which represents an MILP . It does so using a node selection policy; common choices include depth- and best-first search. Then, it chooses an index using a variable selection policy. It next branches on : it sets the left child of to be that same integer program, but with the additional constraint that , and it sets the right child of to be that same integer program, but with the additional constraint that . The algorithm fathoms a leaf, which means that it never will branch on that leaf, if it can guarantee that the optimal solution does not lie along that path. The algorithm terminates when it has fathomed every leaf. At that point, we can guarantee that the best solution to found so far is optimal. See the paper by Balcan et al. [8] for more details.
Balcan et al. [8] study mixed integer linear programs (MILPs) where the goal is to maximize an objective function subject to the constraints that and that some of the components of are contained in . Given a MILP , we use the notation to denote an optimal solution to the MILP’s LP relaxation. We denote the optimal objective value to the MILP’s LP relaxation as , which means that .
Branch-and-bound systematically partitions the feasible set in order to find an optimal solution, organizing the partition as a tree. At the root of this tree is the original integer program. Each child represents the simplified integer program obtained by partitioning the feasible set of the problem contained in the parent node. The algorithm prunes a branch if the corresponding subproblem is infeasible or its optimal solution cannot be better than the best one discovered so far. Oftentimes, branch-and-bound partitions the feasible set by adding a constraint. For example, if the feasible set is characterized by the constraints and , the algorithm partition the feasible set into one subset where , , and , and another where , , and . In this case, we say that the algorithm branches on .
Balcan et al. [8] show how to learn variable selection policies. Specifically, they study score-based variable selection policies, defined below.
Definition 6.10** (Score-based variable selection policy [8]).**
Let score be a deterministic function that takes as input a partial search tree , a leaf of that tree, and an index , and returns a real value . For a leaf of a tree , let be the set of variables that have not yet been branched on along the path from the root of to . A score-based variable selection policy selects the variable to branch on at the node .
This type of variable selection policy is widely used [70, 1, 40]. See the paper by Balcan et al. [8] for examples.
Given arbitrary scoring rules , Balcan et al. [8] provide guidance for learning a linear combination that leads to small expected tree sizes. They assume that all aspects of the tree search algorithm except the variable selection policy, such as the node selection policy, are fixed. In their analysis, they prove the following lemma.
Lemma 6.11** ([8]).**
Let be arbitrary scoring rules and let be an arbitrary MILP over binary variables. Suppose we limit B&B to producing search trees of size . There is a set of at most hyperplanes such that for any connected component of , the search tree B&B builds using the scoring rule is invariant across all .
This piecewise structure immediately implies the following guarantee.
Corollary 6.12**.**
Let be arbitrary scoring rules and let be an arbitrary MILP over binary variables. Suppose we limit B&B to producing search trees of size . For each parameter vector , let be the size of the tree, divided by , that B&B builds using the scoring rule given as input. Let be the set of functions mapping MILPs to . The dual class is -piecewise decomposable, where consists of halfspace indicator functions and consists of constant functions .
Corollary 6.12 and Lemma 3.10 imply the following pseudo-dimension bound.
Corollary 6.13**.**
Let be arbitrary scoring rules and let be an arbitrary MILP over binary variables. Suppose we limit B&B to producing search trees of size . For each parameter vector , let be the size of the tree, divided by , that B&B builds using the scoring rule given as input. Let be the set of functions mapping MILPs to . Then .
Corollary 6.13 matches the pseudo-dimension guarantee that Balcan et al. [8] prove.
Integer quadratic programming.
A diverse array of NP-hard problems, including max-2SAT, max-cut, and correlation clustering, can be characterized as integer quadratic programs (IQPs). An IQP is represented by a matrix . The goal is to find a set maximizing . The most-studied IQP approximation algorithms operate via an SDP relaxation:
[TABLE]
The approximation algorithm must transform, or “round,” the unit vectors into a binary assignment of the variables . In the seminal GW algorithm [41], the algorithm projects the unit vectors onto a random vector , which it draws from the -dimensional Gaussian distribution, which we denote using . If , it sets . Otherwise, it sets .
The GW algorithm’s approximation ratio can sometimes be improved if the algorithm probabilistically assigns the binary variables. In the final step, the algorithm can use any rounding function to set with probability and with probability . See Algorithm 1 for the pseudocode.
Algorithm 1 is known as a Random Projection, Randomized Rounding (RPR2) algorithm, so named by the seminal work of Feige and Langberg [33].
Balcan et al. [7] analyze -linear rounding functions [33] , parameterized by , defined as follows:
[TABLE]
The goal is to learn a parameter such that in expectation, is maximized. The expectation is over several sources of randomness: first, the distribution over matrices ; second, the vector ; and third, the assignment of . This final assignment depends on the parameter , the matrix , and the vector . Balcan et al. [7] refer to this value as the true utility of the parameter . Note that the distribution over matrices, which defines the algorithm’s input, is unknown and external to the algorithm, whereas the Gaussian distribution over vectors as well as the distribution defining the variable assignment are internal to the algorithm.
The distribution over matrices is unknown, so we cannot know any parameter’s true utility. Therefore, to learn a good parameter , we must use samples. Balcan et al. [7] suggest drawing samples from two sources of randomness: the distributions over vectors and matrices. In other words, they suggest drawing a set of samples Given these samples, Balcan et al. [7] define a parameter’s empirical utility to be the expected objective value of the solution Algorithm 1 returns given input , using the vector and in Step 4, on average over all . Generally speaking, Balcan et al. [7] suggest sampling the first two randomness sources in order to isolate the third randomness source. They argue that this third source of randomness has an expectation that is simple to analyze. Using pseudo-dimension, they prove that every parameter , its empirical and true utilities converge.
A bit more formally, Balcan et al. [7] use the notation to denote the distribution that the binary value is drawn from when Algorithm 1 is given as input and uses the rounding function and the hyperplane in Step 4. Using this notation, the parameter has a true utility of
[TABLE]
We also use the notation to denote the expected objective value of the solution Algorithm 1 returns given input , using the vector and in Step 4. The expectation is over the final assignment of each variable . Specifically, . By definition, a parameter’s true utility equals . Given a set , a parameter’s empirical utility is .
Both we and Balcan et al. [7] bound the pseudo-dimension of the function class . Balcan et al. [7] prove that the functions in are piecewise structured: roughly speaking, for a fixed matrix and vector , each function in is a piecewise, inverse-quadratic function of the parameter . To present this lemma, we use the following notation: given a tuple , let be defined such that .
Lemma 6.14** ([7]).**
For any matrix and vector , the function is made up of piecewise components of the form for some . Moreover, if the border between two components falls at some , then it must be that for some in the optimal SDP embedding of .
This piecewise structure immediately implies the following corollary about the dual class .
Corollary 6.15**.**
Let be the set of functions . The dual class is -piecewise decomposable, where consists of threshold functions and consists of inverse-quadratic functions .
Lemma 3.9 and Corollary 6.15 imply the following pseudo-dimension bound.
Corollary 6.16**.**
Let be the set of functions . The pseudo-dimension of is at most .
Corollary 6.16 matches the pseudo-dimension bound that Balcan et al. [7] prove.
6.3 Greedy algorithms
Gupta and Roughgarden [48] provide pseudo-dimension bounds for greedy algorithm configuration, analyzing two canonical combinatorial problems: the maximum weight independent set problem and the knapsack problem. We recover their bounds in both cases.
Maximum weight independent set (MWIS)
In the MWIS problem, the input is a graph with a weight per vertex . The objective is to find a maximum-weight (or independent) set of non-adjacent vertices. On each iteration, the classic greedy algorithm adds the vertex that maximizes to the set. It then removes and its neighbors from the graph. Given a parameter , Gupta and Roughgarden [48] propose the greedy heuristic . In this context, the utility function equals the weight of the vertices in the set returned by the algorithm parameterized by . Gupta and Roughgarden [48] implicitly prove the following lemma about each function (made explicit in work by Balcan et al. [9]). To present this lemma, we use the following notation: let be defined such that .
Lemma 6.17** ([48]).**
For any weighted graph , the function is piecewise constant with at most discontinuities.
This structure immediately implies that the function class has a piecewise-structured dual class.
Corollary 6.18**.**
Let be the set of functions . The dual class is -piecewise decomposable, where consists of threshold functions and consists of constant functions .
Lemma 3.9 and Corollary 6.18 imply the following pseudo-dimension bound.
Corollary 6.19**.**
Let be the set of functions . The pseudo-dimension of is .
This matches the pseudo-dimension bound by Gupta and Roughgarden [48].
Knapsack.
Moving to the classic knapsack problem, the input is a knapsack capacity and a set of items each with a value and a size . The goal is to determine a set with maximium total value such that . Gupta and Roughgarden [48] suggest the family of algorithms parameterized by where each algorithm returns the better of the following two solutions:
- •
Greedily pack items in order of nonincreasing value subject to feasibility.
- •
Greedily pack items in order of subject to feasibility.
It is well-known that the algorithm with achieves a 2-approximation. We use the notation to denote the total value of the items returned by the algorithm parameterized by given input .
Gupta and Roughgarden [48] implicitly prove the following fact about the functions (made explicit in work by Balcan et al. [9]). To present this lemma, we use the following notation: given a tuple , let be defined such that .
Lemma 6.20** ([48]).**
For any tuple , the function is piecewise constant with at most discontinuities.
This structure immediately implies that the function class has a piecewise-structured dual class.
Corollary 6.21**.**
Let be the set of functions . The dual class is -piecewise decomposable, where consists of threshold functions and consists of constant functions .
Lemma 3.9 and Corollary 6.21 imply the following pseudo-dimension bound.
Corollary 6.22**.**
Let be the set of functions . The pseudo-dimension of is .
This matches the pseudo-dimension bound by Gupta and Roughgarden [48].
6.4 Revenue maximization
The design of revenue-maximizing multi-item mechanisms is a notoriously challenging problem. Remarkably, the revenue-maximizing mechanism is not known even when there are just two items for sale. In this setting, the mechanism designer’s goal is to field a mechanism with high expected revenue on the distribution over agents’ values. A line of research has provided generalization guarantees for mechanism design in the context of revenue maximization [29, 23, 25, 43, 47, 19, 44, 77, 76, 6]. These papers focus on sales settings: there is a seller, not included among the agents, who will use a mechanism to allocate a set of goods among the agents. The agents submit bids describing their values for the goods for sale. The mechanism determines which agents receive which items and how much the agents pay. The seller’s revenue is the sum of the agents’ payments. The mechanism designer’s goal is to select a mechanism that maximizes the revenue. In contrast to the mechanisms we analyze in Section 5, these papers study mechanisms that are not necessarily budget-balanced. Specifically, under every mechanism they study, the sum of the agents’ payments—the revenue—is at least zero. As in Section 5, all of the mechanisms they analyze are incentive compatible.
We study the problem of selling heterogeneous goods to buyers. They denote a bundle of goods as a subset . Each buyer has a valuation function over bundles of goods. In this setting, the set of problem instances consists of -tuples of buyer values . As in Section 5, selling mechanisms defined by an allocation function and a set of payment functions. Every auction in the classes they study is incentive compatible, so they assume that the bids equal the bidders’ valuations. An allocation function maps the values to a division of the goods , where is the set of goods buyer receives. For each agent , there is a payment function which maps values to a payment that agent must make.
We recover generalization guarantees proved by Balcan et al. [10] which apply to a variety of widely-studied parameterized mechanism classes, including posted-price mechanisms, multi-part tariffs, second-price auctions with reserves, affine maximizer auctions, virtual valuations combinatorial auctions mixed-bundling auctions, and randomized mechanisms. They provided new bounds for mechanism classes not previously studied in the sample-based mechanism design literature and matched or improved over the best known guarantees for many classes. They proved these guarantees by uncovering structure shared by all of these mechanisms: for any set of buyers’ values, revenue is a piecewise-linear function of the mechanism’s parameters. This structure is captured by our definition of piecewise decomposability.
Each of these mechanism classes is parameterized by a -dimensional vector for some . For example, when , might be a vector of prices for each of the items. The revenue of a mechanism is the sum of the agents’ payments. Given a mechanism parameterized by a vector , we denote the revenue as , where .
Balcan et al. [10] provide psuedo-dimension bounds for any mechanism class that is delineable. To define this notion, for any fixed valuation vector , we use the notation to denote revenue as a function of the mechanism’s parameters.
Definition 6.23** (-delineable [10]).**
A mechanism class is -delineable if:
The class consists of mechanisms parameterized by vectors from a set ; and 2. 2.
For any , there is a set of hyperplanes such that for any connected component of , the function is linear over
Delineability naturally translates to decomposability, as we formalize below.
Lemma 6.24**.**
Let be a set of revenue functions corresponding to a -delineable mechanism class. The dual class is -piecewise decomposable, where consists of halfspace indicator functions and consists of linear functions .
Lemmas 3.10 and 6.24 imply the following bound.
Corollary 6.25**.**
Let be a set of revenue functions corresponding to a -delineable mechanism class. The pseudo-dimension of is at most .
Corollary 6.25 matches the pseudo-dimension bound that Balcan et al. [10] prove.
7 Experiments
We complement our theoretical guarantees with experiments in several settings to help illustrate our bounds. Our experiments are in the contexts of both new and previously-studied domains: tuning parameters of sequence alignment algorithms in computational biology, tuning parameters of sales mechanisms so as to maximize revenue, and tuning parameters of voting mechanisms so as to maximize social good. We summarize two observations from the experiments that help illustrate the theoretical message of this paper.
Observation 1*.*
First, using sequence alignment data (Section 7.1), we demonstrate that with a finite number of samples, an algorithm’s average performance over the samples provides a tight estimate of its expected performance on unseen instances.
Observation 2*.*
Second, our experiments empirically illustrate that two algorithm families for the same computational problem may require markedly different training set sizes to avoid overfitting, a fact that has also been explored in prior theoretical research [7, 10]. This shows that it is crucial to bound an algorithm family’s intrinsic complexity to provide accurate guarantees, and in this paper we provide tools for doing so. Our experiments here are in the contexts of mechanism design for revenue maximization (Section 7.2.1) and mechanism design for voting (Section 7.2.2), both using real-world data. In both of these settings, we analyze two natural, practical mechanism families where one of the families is intrinsically simple and the other is intrinsically complex. When we use Theorem 3.3 to select enough samples to ensure that overfitting does not occur for the simple class, we indeed empirically observe overfitting when optimizing over the complex class. The complex class requires more samples to avoid overfitting. In revenue-maximizing mechanism design, it was known that there exists a separation between the intrinsic complexities of the two corresponding mechanism families [10], though this was not known in voting mechanism design. In both of these settings, despite surface-level similarities between the complex and simple mechanism families, there is a pronounced difference in their intrinsic complexities, as illustrated by these experiments.
7.1 Sequence alignment experiments
Changing the alignment parameter can alter the accuracy of the produced alignments. Figure 5 shows the regions of the gap-open/gap-extension penalty plane divided into regions such that each region corresponds to a different computed alignment. The regions in the figure are produced using the XPARAL software of Gusfield and Stelling [49], with using the BLOSUM62 amino acid replacement matrix, the scores in each region were computed using Robert Edgar’s qscore package333http://drive5.com/qscore/. The alignment sequences are a single pairwise alignment from the data set described below.
To illustrate Observation 1, we use the IPA tool [60] to learn optimal parameter choices for a given set of example pairwise sequence alignments. We used 861 protein multiple sequence alignment benchmarks that had been previously been used in DeBlasio and Kececioglu [24], which split these benchmarks into 12 cross-validation folds that evenly distributed the “difficulty” of an alignment (the accuracy of the alignment produced using aligner defaults parameter choice). All pairwise alignments were extracted from each multiple sequence alignment. We then took randomized increasing sized subsets of the pairwise sequence alignments from each training set and found the optimal parameter choices for affine gap costs and alphabet-dependent substitution costs. These parameters were then given to the Opal aligner [v3.1b, 105] to realign the pairwise alignments in the associated test sets.
Figure 6 shows the impact of increasing the number of training examples used to optimize parameter choices. As the number of training examples increases, the optimized parameter choice is less able to fit the training data exactly and thus the training accuracy decreases. For the same reason the parameter choices are more general and the test accuracy increases. The test and training accuracies are roughly equal when the training set size is close to examples and remains equal for larger training sets. The test accuracy is actually slightly higher and this is likely due to the training subset not representing the distribution of inputs as well as the full test set due to the randomization being on all of the alignments rather than across difficulty, as was done to create the cross-validation separations.
7.2 Mechanism design experiments
In this section, we provide experiments analyzing the estimation error of several mechanism classes. We introduced the notion of estimation error in Section 2, but review it briefly here. For a class of mechanisms parameterized by vectors (such as the class of neutral affine maximizers from Section 5), let be the utility of the mechanism parameterized by when the agents’ values are defined by the vector . For example, this utility might equal its social welfare or revenue. Given a set of samples , the mechanism class’s estimation error equals the maximum difference, across all parameter vectors , between the average utility of the mechanism over the samples and its expected utility . We know that with high probability, the estimation error is bounded by , where is the pseudo-dimension of the mechanism class. In other words, for all parameter vectors ,
[TABLE]
Said another way, samples are sufficient to ensure that with high probability, for every parameter vector , the difference between the average and expected utility of the mechanism parameterized by is at most .
7.2.1 Revenue maximization for selling mechanisms
We provide experiments that illustrate our guarantees in the context of revenue maximization (which we introduced in Section 6.4). The type of mechanisms we analyze in our experiments are second-price auctions with reserves, one of the most well-studied mechanism classes in economics [98]. In this setting, there is one item for sale and a set of agents who are interested in buying that item. This mechanism family is defined by a parameter vector , where each entry is the reserve price for agent . The agents report their values for the item to the auctioneer. If the highest bidder—say, agent —reports a bid that is larger than her reserve , she wins the item and pays the maximum of the second-highest bid and her reserve . Otherwise, no agent wins the item. The second-price auction (SPA) is called anonymous if every agent has the same reserve price: . Otherwise, the SPA is called non-anonymous. Like neutral affine maximizers, SPAs are incentive compatible, so we assume the agents’ bids equal their true values. We refer to the class of non-anonymous SPAs as and the class of anonymous SPAs as .
The class of non-anonymous SPAs is more complex than the class of anonymous SPAs , and thus the sample complexity of the former is higher than the latter. However, non-anonymous SPAs can provide much higher revenue than anonymous SPAs. We illustrate this tradeoff in our experiments, which helps exemplify Observation 2.
In a bit more detail, the pseudo-dimension of the class of non-anonymous SPAs is , an upper bound proved in prior research [10, 77] which can be recovered using the techniques in this paper, as we summarize in Section 6.4. What’s more, Balcan et al. [10] proved a pseudo-dimension lower bound of for this class. Meanwhile, the pseudo-dimension of the class of anonymous SPAs is . This upper bound was proved in prior research by Morgenstern and Roughgarden [77] and Balcan et al. [10], and it follows from the general theory presented in this paper, as we formalize below in Lemma 7.1. In our experiments, we exhibit a distribution over agents’ values under which the following properties hold:
The true estimation error of the set of non-anonymous SPAs is larger than our upper bound on the worst-case estimation error of the set of anonymous SPAs . 2. 2.
The expected revenue of the optimal non-anonymous SPA is significantly larger than the expected revenue of the optimal anonymous SPA: the former is 0.38 and the latter is 0.57.
These two points imply that even though it may be tempting to optimize over the class of non-anonymous SPAs in order to obtain higher expected revenue, the training set must be large enough to ensure the empirically-optimal mechanism has nearly optimal expected revenue.
The distribution we analyze is over the Jester Collaborative Filtering Dataset [42], which consists of ratings from tens of thousands of users of one hundred jokes—in this example, the jokes could be proxies for comedians, and the agents could be venues who are bidding for the opportunity to host the comedians. Since the auctions we analyze in these experiments are for a single item, we run experiments using agents’ values for a single joke, which we select to have a roughly equal number of agents with high values as with low values (we describe this selection in greater detail below). Figure 7 illustrates the outcome of our experiments.
The orange dashed line is our upper bound on the estimation error of the class of anonymous SPAs , which equals , with . This upper bound has been presented in prior research [77, 10], and we recover it using the general theory presented in this paper, as we demonstrate below in Lemma 7.1. The blue solid line is the true estimation error444Sample complexity and estimation error bounds for SPAs have been studied in prior research [77, 25, 10], and our guarantees match the bounds provided in that literature. of the class of non-anonymous SPAs over the Jester dataset, which we calculate via the following experiment. For a selection of training set sizes , we draw sample agent values, with the number of agents equal to 10,612 (as we explain below). We then find a vector of non-anonymous reserves with maximum average revenue over the samples but low expected revenue on the true distribution, as we describe in greater detail below. We plot this difference between average and expected revenue, averaged over 100 trials of the same procedure. As these experiments illustrate, there is a tradeoff between the sample complexity and revenue guarantees of these two classes, and it is crucial to calculate a class’s intrinsic complexity to provide accurate guarantees. We now explain the details of these experiments.
Distribution over agents’ values.
We use the Jester Collaborative Filtering Dataset [42], which consists of ratings from 24,983 users of 100 jokes. The users’ ratings are in the range , so we normalize their values to lie in the interval . We select a joke which has at least 5000 (normalized) ratings in the interval and at least 5000 (normalized) ratings in the interval (specifically, Joke #22). There is a total of 5334 ratings of the first type and 5278 ratings of the second type. Let be all 10,612 values. For every , we define the following valuation vector : agent ’s value equals and for all other agents , . We define the distribution over agents’ values to be the uniform distribution over .
Empirically-optimal non-anonymous reserve vectors with poor generalization.
Given a set of samples , let define the empirical distribution over (that is, equals the number of times appears in divided by ). Then for any non-anonymous reserve vector , the average revenue over the samples is
[TABLE]
This is because for every vector , the only agent with a non-zero value is agent , whose value is . Therefore, the highest bidder is agent , who wins the item if , and pays reserve , which is the maximum of and the second-highest bid. As is clear from Equation (21), if , an empirically-optimal reserve vector (which maximizes average revenue over the samples) will set , and if , can be set arbitrarily, because it has no effect on the average revenue over the samples. In our experiments, for all , we set . The intuition is that those agents with and will not win the item, and therefore will drag down expected revenue.
In Figure 7, the orange dashed line is the difference between the average value of over and its expected value, which we calculate via the following experiment. For a selection of training set sizes , we draw the set . We then construct the reserve vector as described above. We plot the difference between the average value of over and its expected value, averaged over 100 trials of the same procedure.
Estimation error of anonymous SPAs.
We now use our general theory to bound the estimation error of the class of anonymous SPAs so that we can plot the blue solid line in Figure 7. This pseudo-dimension upper bound has been presented in prior research [77, 10], but here we show that it can be recovered using this paper’s techniques. We use Lemma 3.8 to obtain a slightly tighter pseudo-dimension bound (up to constant factors) than that of Corollary 6.25.
Given a reserve price and valuation vector , let be the revenue of the anonymous second-price auction with a reserve price of when the bids equal . Using Lemma 3.8, we prove that the pseudo-dimension of this class of revenue functions is at most 2.
Lemma 7.1**.**
The pseudo-dimension of the class is at most 2.
Proof.
Given a vector of bids, let be the highest bid in and let be the second-highest bid. Under the anonymous SPA, the highest bidder wins the item if and pays . Therefore By definition of the dual function,
[TABLE]
The dual function is thus an increasing function of in the interval and is equal to zero in the interval . Therefore, the function has at most two oscillations (as in Definition 3.7). By Lemma 3.8, the pseudo-dimension of is at most the largest integer such that , which equals 2. Therefore, the theorem statement holds. ∎
By Equation (2), with probability over the draw of , for any reserve ,
[TABLE]
In Figure 7, the blue solid line equals the right-hand-side of Equation (22) with as a function of .
Discussion.
In summary, this section exemplifies a distribution over agents’ values where:
The true estimation error of the set of non-anonymous SPAs (the orange dashed line in Figure 7) is larger than our upper bound on the worst-case estimation error of the set of anonymous SPAs (the blue solid line in Figure 7). 2. 2.
The expected revenue of the optimal non-anonymous SPA is significantly larger than the expected revenue of the optimal anonymous SPA: the former is 0.38 and the latter is 0.57.
Therefore, there is a tradeoff between the sample complexity and revenue guarantees of these two classes.
7.2.2 Social welfare maximization for voting mechanisms
In this section, we provide similar experiments as those in the previous section, but in the context of neutral affine maximizers (NAMs), which we defined in Section 5. We present a simple subset of neutral affine maximizers (NAMs) with a small estimation error upper bound. We then experimentally demonstrate that the true estimation error of the class of NAMs is larger than this simple subset’s estimation error. Therefore, it is crucial to calculate a class’s intrinsic complexity in order to provide accurate guarantees. These experiments further illustrate Observation 2.
Our simple set of mechanisms is defined as follows. One agent is selected to be the sink agent (a sink agent has the weight ), and every other agent’s weight is set to 1. In other words, this class is defined by the set of all vectors where exactly one component of is equal to zero. We use the notation to denote this simple class, to denote the set of all NAMs and to denote the social welfare of the NAM parameterized by given the valuation vector .
Since there are NAMs in , a Hoeffding and union bound tells us that for any distribution , with probability over the draw of valuation vectors , for all parameter vectors ,
[TABLE]
This is the orange dashed line in Figure 8. Meanwhile, as we prove in Section 5, the pseudo-dimension of the class of all NAMs is , so it is a more complex set of mechanisms than . To experimentally compute the lower bound on the true estimation error of the class of all NAMs , we identify a distribution such that the class has high estimation error, as we describe below.
Distribution.
As in the previous section, we use the Jester Collaborative Filtering Dataset [42] to define our distribution. This dataset consists of ratings from 24,983 users of 100 jokes. The users’ ratings are in the range , so we normalize their ratings to lie in the interval . We begin by selecting two jokes (jokes #7 and #15) such that—at a high level—a large number of agents either like the first joke a lot and do not like the second joke, or do not like the first joke and like the second joke a medium amount. We explain the intuition behind this choice below. Specifically, we split the agents into two groups: in the first group, the agents rated joke 1 at least 0.35 and rated joke 2 at most 0, and in the second group, the agents rated joke 1 at most 0 and rated joke 2 between 0 and 0.15. We call the set of ratings corresponding to the first group and those corresponding to the second group . The set has size 870 and has size 1677.
We use and to define a distribution over the valuations of agents for two jokes. The support of consists of 500 valuation vectors . For , is defined as follows. The values of agent for the two jokes, , are chosen uniformly at random from and the values of agent are chosen uniformly at random from . Every other agent has a value of .
Parameter vector with poor estimation error.
Given a set of samples , we define a parameter vector with high estimation error as follows: for all ,
[TABLE]
Intuitively, this parameter vector555Although this parameter vector has high average social welfare over the samples, it may set multiple agents to be sink agents, which may be wasteful. We leave the problem of finding a parameter vector with high estimation error and only a single sink agent to future research. has high estimation error for the following reason. Suppose . The only agents with nonzero values in are agent and agent . Since , and . Therefore, agent ’s favorite joke is selected. Since agent ’s values are from the set , they have a value of at least 0.35 for joke 1 and a value of at most 0 for joke 2. Therefore, joke 1 will be the selected joke. Meanwhile, by the same reasoning for every , if we run the NAM defined by , joke 2 will be the selected joke. In expectation over , joke 1 has a significantly higher social welfare than joke 2. Therefore, the NAM defined by will have a high average social welfare over the samples in but a low expected social welfare, which means it will have high estimation error. We illustrate this intuition in our experiments.
Experimental procedure.
We repeat the following experiment 100 times. For various choices of , we draw a set of samples , compute the parameter vector defined by Equation (24), and compute the difference between the average social welfare of over and its expected social welfare. We plot the difference averaged over all 100 runs.
Discussion.
These experiments demonstrate that although the simple and complex NAM families and are artificially similar (they are both defined by the agent weights), the complex family requires far more samples to avoid overfitting than the simple family . This illustrates the importance of using our pseudo-dimension bounds to provide accurate guarantees.
8 Conclusions
We provided a general sample complexity theorem for learning high-performing algorithm configurations. Our bound applies whenever a parameterized algorithm’s performance is a piecewise-structured function of its parameters: for any fixed problem instance, boundary functions partition the parameters into regions where performance is a well-structured function. We proved this guarantee by exploiting intricate connections between primal function classes (measuring the algorithm’s performance as a function of its input) and dual function classes (measuring the algorithm’s performance on a fixed input as a function of its parameters). We demonstrated that many parameterized algorithms exhibit this structure and thus our main theorem implies sample complexity guarantees for a broad array of algorithms and application domains.
A great direction for future research is to build on these ideas for the sake of learning a portfolio of configurations, rather than a single high-performing configuration. At runtime, machine learning is used to determine which configuration in the portfolio to employ for the given input. Gupta and Roughgarden [48] and Balcan et al. [14] have provided initial results in this direction, but a general theory of portfolio-based algorithm configuration is yet to be developed.
Acknowledgments.
This research is funded in part by the Gordon and Betty Moore Foundation’s Data-Driven Discovery Initiative (GBMF4554 to C.K.), the US National Institutes of Health (R01GM122935 to C.K.), the US National Science Foundation (a Graduate Research Fellowship to E.V. and grants IIS-1901403 to M.B. and T.S., IIS-1618714, CCF-1535967, CCF-1910321, and SES-1919453 to M.B., IIS-1718457, IIS-1617590, and CCF-1733556 to T.S., and DBI-1937540 to C.K.), the US Army Research Office (W911NF-17-1-0082 and W911NF2010081 to T.S.), the Defense Advanced Research Projects Agency under cooperative agreement HR00112020003 to M.B., an AWS Machine Learning Research Award to M.B., an Amazon Research Award to M.B., a Microsoft Research Faculty Fellowship to M.B., a Bloomberg Research Grant to M.B., a fellowship from Carnegie Mellon University’s Center for Machine Learning and Health to E.V., and by the generosity of Eric and Wendy Schmidt by recommendation of the Schmidt Futures program.
Appendix A Helpful lemmas
Lemma A.1** (Shalev-Shwartz and Ben-David [92]).**
Let and . If , then .
The following is a corollary of Rolle’s theorem that we use in the proof of Lemma 4.6.
Lemma A.2** (Tossavainen [96]).**
Let be a polynomial-exponential sum of the form , where and . The number of roots of is upper bounded by .
Corollary A.3**.**
Let be a polynomial-exponential sum of the form
[TABLE]
where and . The number of roots of is upper bounded by .
Proof.
Note that if and only if
[TABLE]
Therefore, the corollary follows from Lemma A.2. ∎
Appendix B Additional details about our general theorem
See 3.10
Proof.
First, we prove that the VC-dimension of the dual class is at most . The dual class consists of functions for all where . Let be the class of halfspace thresholds . It is well-known that , which we prove means that . For a contradiction, suppose can shatter functions . Then for every subset , there exists a parameter vector such that if and only if . This means that can shatter the tuples as well, which contradicts the fact that . Therefore, .
By a similar argument, we prove that the pseudo-dimension of the dual class is at most . The dual class consists of functions for all where . Let be the class of linear functions . It is well-known that , which we prove means that . For a contradiction, suppose can shatter functions . Then there exist witnesses such that for every subset , there exists a parameter vector such that if and only if . This means that can shatter the tuples as well, which contradicts the fact that . Therefore, .
The lemma statement now follows from Theorem 3.3. ∎
Appendix C Additional details about sequence alignment (Section 4.1)
Lemma C.1**.**
Fix a pair of sequences . There are at most alignments of and .
Proof.
For any alignment , we know that and for all , if , then and vice versa. This means that and have the same number of gaps. To prove the upper bound, we count the number of alignments where and each have exactly gaps. There are choices for the sequence . Given a sequence , we can only pair a gap in with a non-gap in . Since there are gaps in and non-gaps in , there are choices for the sequence once is fixed. This means that there are alignments where and each have exactly gaps. Summing over , the total number of alignments is at most . ∎
See 4.3
Proof.
To prove this theorem, we identify:
An alphabet , 2. 2.
A set of sequence pairs , 3. 3.
A ground-truth alignment for each sequence pair , and 4. 4.
A set of witnesses such that for any subset , there exists an indel penalty parameter such that if , then and if , then .
We now describe each of these four elements in turn.
The alphabet .
Let . The alphabet consists of characters666To simplify the proof, we use this alphabet of size , but we believe it is possible to adapt this proof to handle the case where there are only 4 characters in the alphabet. we denote as .
The set of sequence pairs.
These sequence pairs are defined by a set of subsequence pairs . Each pair is defined as follows:
- •
The subsequence begins with s followed by . For example, .
- •
The subsequence begins with 1 , followed by s, followed by 1 . For example, .
Therefore, and are both of length .
We use these subsequence pairs to define a set of sequence pairs. The first sequence pair, is defined as follows: is the concatenation of all subsequences and is the concatenation of all subsequences :
[TABLE]
Next, and are the concatenation of every subsequence:
[TABLE]
Similarly, and are the concatenation of every subsequence:
[TABLE]
Generally speaking, and are the concatenation of every subsequence:
[TABLE]
To explain the index of the last subsequence of every pair, since is a power of two, we know that is divisible by 2, is divisible by 4, and more generally, is divisible by .
We claim that there are a total of such sequence pairs. To see why, note that each sequence in the first pair and consists of subsequences. Each sequence in the second pair and consists of subsequences. More generally, each sequence in the pair and consists of subsequences. The final pair will will consist of only one subsequence, so , or in other words .
We also claim that each sequence has length at most . This is because the longest sequence pair is the first, . By definition of the subsequences , these two sequences are of length Therefore, all sequence pairs are of length at most .
Example C.2**.**
Suppose that . Then and . The three sequence pairs have the following form777The maximum length of these six strings is 42, which is smaller than 128, as required.:
[TABLE]
A ground-truth alignment for every sequence pair.
To define a ground-truth alignment for all sequence pairs, we first define two alignments per subsequence pair . The resulting ground-truth alignments will be a concatenation of these alignments. The first alignment, which we denote as , is defined as follows: begins with s, followed by 1 , followed by gap characters, followed by 1 ; begins with gap characters, followed by 1 , followed by s, followed by 1 . For example,
[TABLE]
The second alignment, which we denote as , is defined as follows: begins with s, followed by 1 , followed by gap characters, followed by 1 ; begins with 1 , followed by gap characters, followed by s, followed by 1 . For example,
[TABLE]
We now use these alignments to define a ground-truth alignment per sequence pair . Beginning with the first pair , where
[TABLE]
we define the alignment of to be and we define the alignment of to be . Moving on to the second pair , where
[TABLE]
we define the alignment of to be and we define the alignment of to be . Generally speaking, each pair , where
[TABLE]
is made up of subsequences. Since is a power of two, this number of subsequences is odd. We define the alignment of to alternate between alignments of type and alignments of type , beginning and ending with alignments of the first type. Specifically, the alignment is . Similarly, we define the alignment of to be
[TABLE]
The witnesses.
We define the values that witness the shattering of these sequence pairs to be .
Shattering the sequence pairs.
Our goal is to show that for any subset , there exists an indel penalty parameter such that if , then and if , then . To prove this, we will use two helpful claims, Claims C.3 and C.4.
Claim C.3**.**
For any pair and indel parameter , there exists an alignment
[TABLE]
such that each character in is matched to in .
Proof.
Let be an alignment such that some character in is not matched to in . Denote the alignment as . Let be the smallest integer such that for some indices , and . Next, let be the maximum index smaller than and such that for some . We illustrate and in Figure 9(a).
By definition of , we know that . We also know there is at least one gap character in because otherwise, the characters would be aligned in . Moreover, we know there is at most one match among these elements between characters other than (namely, between the character ). If we rearrange all of these gap characters so that they fall directly after in both sequences, as in Figure 9(b), then we may lose the match between the character , but we will gain the match between the character . Moreover, the number of indels remains the same, and all matches in the remainder of the alignment will remain unchanged. Therefore, this rearranged alignment has at least as high an objective function value as , so the claim holds. ∎
Based on this claim, we will assume, without loss of generality, that for any pair and indel parameter , under the alignment returned by the algorithm , all characters in are matched to in .
Claim C.4**.**
Suppose that the character is in and . The characters will be matched in if and only if .
Proof.
Since all characters are matched in , in order to match , it is necessary to add exactly gap characters: all and characters must be matched with gap characters. Under the objective function , this one match will be worth the penalty if and only if , as claimed. ∎
We now use Claims C.3 and C.4 to prove that we can shatter the sequence pairs , …, .
Claim C.5**.**
There are thresholds such that as ranges from 0 to 1, when crosses one of these thresholds, switches from above to below , or vice versa, beginning with and ending with .
Proof.
Recall that
[TABLE]
so in and , the characters are . Also, the reference alignment of is and the reference alignment of is
[TABLE]
We know that when the indel penalty is equal to zero, all characters will be aligned, as will all characters. This means we will correctly align all characters and we will correctly align all characters in the pairs, but we will incorrectly align the characters in the pairs. The number of pairs in this reference alignment is and the number of pairs is . Therefore, the utility of the alignment that maximizes the number of matches equals the following:
[TABLE]
where the final inequality holds because .
Next, suppose we increase to lie in the interval . Since it is no longer the case that , we know that the characters will no longer be matched, and thus we will correctly align this character according to the reference alignment. This means we will correctly align all characters and we will correctly align all characters in the pairs, but we will incorrectly align all but one of the characters in the pairs. Therefore, the utility of the alignment that maximizes is
[TABLE]
where the final inequality holds because .
Next, suppose we increase to lie in the interval . Since it is no longer the case that , we know that the characters will no longer be matched, and thus we will incorrectly align this character according to the reference alignment. This means we will correctly align all characters and we will correctly align the characters in all but one of the pairs, but we will incorrectly align all but one of the characters in the pairs. Therefore, the utility of the alignment that maximizes is
[TABLE]
In a similar fashion, every time crosses one of the thresholds , the utility will shift from above to below or vice versa, as claimed. ∎
The above claim demonstrates that the sequence pairs are shattered, each with the witness . After all, for every and every interval where is uniformly above or below , there exists a subpartition of this interval into the two intervals
[TABLE]
such that in the first interval, and in the second, . Therefore, for any subset , there exists an indel penalty parameter such that if , then and if , then . ∎
C.1 Tighter guarantees for a structured algorithm subclass: sequence alignment using hidden Markov models
While we focused on the affine gap model in the previous section, which was inspired by the results in Gusfield, Balasubramanian, and Naor [50], the result in Pachter and Sturmfels [81] helps to provide uniform convergence guarantees for any alignment scoring function that can be modeled as a hidden Markov model (HMM). A bound on the number of parameter choices that emit distinct sets of co-optimal alignments in that work is found by taking an algebraic view of the alignment HMM with tunable parameters. In fact, the bounds provided can be used to provide guarantees for many types of HMMs.
Lemma C.6**.**
Let be a set of co-optimal-constant algorithms and let be a utility function mapping tuples of sequence pairs and alignments to the interval . Let be the set of functions mapping sequence pairs to . For some constant , the dual class is -piecewise decomposable, where consists of halfspace indicator functions and consists of constant functions .
Proof.
Fix a sequence pair and and consider the function from the dual class , where . Consider the set of alignments . There are at most sets of co-optimal solutions as we range over [81]. The remainder of the proof is analogous to that for Lemma 4.2. ∎
Finally the results of Lemma C.6 imply the following pseudo-dimension bound.
Corollary C.7**.**
Let be a set of co-optimal-constant algorithms and let be a utility function mapping tuples to . Let be the set of functions
[TABLE]
mapping sequence pairs to . Then
C.2 Additional details about progressive multiple sequence alignment (Section 4.1.3)
Definition C.8**.**
Let be a sequence alignment. The consensus sequence of this alignment is the sequence where is the most-frequent non-gap character in the position in the alignment (breaking ties in a fixed but arbitrary way). For example, the consensus sequence of the alignment
[TABLE]
is ATCC when ties are broken in favor of A over G.
Figure 10 illustrates an example of this algorithm in action, and corresponds to the psuedo-code given in Algorithm 2. The first loop matches with Figure 10(a), the second and third match with Figure 10(b).
See 4.4
Proof.
A key step in the proof of Lemma 4.1 for pairwise alignment shows that for any pair of sequences , we can find a set of hyperplanes such that for any pair and belonging to the same connected component of , we have . We use this result to prove the following claim.
Claim C.9**.**
For each node in the guide tree, there is a set of hyperplanes where for any connected component of , the alignment and consensus sequence computed by is fixed across all . Moreover, the size of is bounded as follows:
[TABLE]
where .
Before we prove Claim C.9, we remark that the longest consensus sequence computed for any node of the guide tree has length at most , which is a bound on the sum of the lengths of the input sequences.
Proof of Claim C.9.
We prove this claim by induction on the guide tree . The base case corresponds to the leaves of . On each leaf, the alignment and consensus sequence constructed by is constant for all , since there is only one string to align (i.e., the input string placed at that leaf). Therefore, the claim holds for the leaves of . Moving to an internal node , suppose that the inductive hypothesis holds for its children and . Assume without loss of generality that , so that . Let and be the sets of hyperplanes corresponding to the children and . By the inductive hypothesis, these sets are each of size at most
[TABLE]
Letting , we are guaranteed that for every connected component of , the alignment and consensus string computed by for both children and is constant. Based on work by Buck [18], we know that there are at most connected components of . For each region, by the same argument as in the proof of Lemma 4.1, there are an additional hyperplanes that partition the region into subregions where the outcome of the pairwise merge at node is constant. Therefore, there is a set of at most
[TABLE]
hyperplanes where for every connected component of , the alignment and consensus string computed by at is invariant. ∎
Applying Claim C.9 to the root of the guide tree, the function is piecewise constant with linear boundary functions. The lemma then follows from the following chain of inequalities:
[TABLE]
∎
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1Achterberg [2009] Tobias Achterberg. SCIP: solving constraint integer programs. Mathematical Programming Computation , 1(1):1–41, 2009.
- 2Alabi et al. [2019] Daniel Alabi, Adam Tauman Kalai, Katrina Ligett, Cameron Musco, Christos Tzamos, and Ellen Vitercik. Learning to prune: Speeding up repeated computations. In Conference on Learning Theory (COLT) , 2019.
- 3Assouad [1983] Patrick Assouad. Densité et dimension. Annales de l’Institut Fourier , 33(3):233–282, 1983.
- 4Balcan [2020] Maria-Florina Balcan. Data-driven algorithm design. In Tim Roughgarden, editor, Beyond Worst Case Analysis of Algorithms . Cambridge University Press, 2020.
- 5Balcan et al. [2005] Maria-Florina Balcan, Avrim Blum, Jason D Hartline, and Yishay Mansour. Mechanism design via machine learning. In Proceedings of the Annual Symposium on Foundations of Computer Science (FOCS) , pages 605–614, 2005.
- 6Balcan et al. [2016] Maria-Florina Balcan, Tuomas Sandholm, and Ellen Vitercik. Sample complexity of automated mechanism design. In Proceedings of the Annual Conference on Neural Information Processing Systems (Neur IPS) , 2016.
- 7Balcan et al. [2017] Maria-Florina Balcan, Vaishnavh Nagarajan, Ellen Vitercik, and Colin White. Learning-theoretic foundations of algorithm configuration for combinatorial partitioning problems. Conference on Learning Theory (COLT) , 2017.
- 8Balcan et al. [2018 a] Maria-Florina Balcan, Travis Dick, Tuomas Sandholm, and Ellen Vitercik. Learning to branch. International Conference on Machine Learning (ICML) , 2018 a.