Learning to Optimize Computational Resources: Frugal Training with Generalization Guarantees

TL;DR

This paper introduces a novel algorithm that learns a finite set of promising parameters from an infinite set, enabling efficient resource optimization with generalization guarantees in configuration problems.

Contribution

It presents a new method for learning promising parameters within infinite sets, addressing limitations of existing discretization approaches.

Findings

The algorithm effectively identifies near-optimal parameters in infinite spaces.

It applies to various domains with piecewise constant performance functions.

Provides theoretical guarantees for generalization and resource efficiency.

Abstract

Algorithms typically come with tunable parameters that have a considerable impact on the computational resources they consume. Too often, practitioners must hand-tune the parameters, a tedious and error-prone task. A recent line of research provides algorithms that return nearly-optimal parameters from within a finite set. These algorithms can be used when the parameter space is infinite by providing as input a random sample of parameters. This data-independent discretization, however, might miss pockets of nearly-optimal parameters: prior research has presented scenarios where the only viable parameters lie within an arbitrarily small region. We provide an algorithm that learns a finite set of promising parameters from within an infinite set. Our algorithm can help compile a configuration portfolio, or it can be used to select the input to a configuration algorithm for finite parameter spaces. Our approach applies to any configuration problem that satisfies a simple yet ubiquitous structure: the algorithm's performance is a piecewise constant function of its parameters. Prior research has exhibited this structure in domains from integer programming to clustering.

Equations

$${\mathbb{E}}_{j\sim \Gamma }\left[\ell \left(\hat{\mathbf{\rho }},j\right)\right]\le (1+ϵ)\underset{\mathbf{\rho }\in \text{pazocal}P}{\inf }\left\{{\mathbb{E}}_{j\sim \Gamma }\left[\ell \left(\mathbf{\rho },j\right)\right]\right\}.$$

$${\mathbb{E}}_{j\sim \Gamma }\left[\min \left\{\ell \left(\hat{\mathbf{\rho }},j\right),t_{\delta }\left(\hat{\mathbf{\rho }}\right)\right\}\right]\le (1+ϵ)\underset{\mathbf{\rho }\in \text{pazocal}P}{\inf }\left\{{\mathbb{E}}_{j\sim \Gamma }\left[\min \left\{\ell \left(\mathbf{\rho },j\right),t_{\delta /2}(\mathbf{\rho })\right\}\right]\right\}.$$

$${\mathbb{E}}_{j\sim \Gamma }\left[\min \left\{\ell \left(\hat{\mathbf{\rho }},j\right),t_{\delta }\left(\hat{\mathbf{\rho }}\right)\right\}\right]$$

$$\le \sqrt{1+ϵ}\cdot {\mathbb{E}}_{j\sim \Gamma }\left[\min \left\{\ell \left({\mathbf{\rho }}^{\prime },j\right),t_{\delta /2}\left({\mathbf{\rho }}^{\prime }\right)\right\}\right]$$

$$\le (1+ϵ)\cdot OP{T}_{\delta /4},$$

$$\left(\text{pazocal}{P}_1,z_1,{\mathbf{\tau }}_1\right),\dots ,\left(\text{pazocal}{P}_k,z_k,{\mathbf{\tau }}_k\right)\in 2^{\text{pazocal}P}\times [0,1]\times {\mathbb{Z}}^{|\text{pazocal}{S}_t|}.$$

$$\eta =\min \left\{\frac{\sqrt[4]{1+ϵ}-1}{8},\frac{1}{9}\right\}\text{and}c=\frac{16\sqrt[4]{1+ϵ}}{\delta }:$$

$$\sum _{t=1}^{\bar{t}}\left|\text{textsc}Partition\left(\text{pazocal}{S}_t,c\cdot OP{T}_{\delta /4}\right)\right|.$$

$$\tilde{O}\left(\frac{d\ln \left|\text{textsc}Partition\left(\text{pazocal}{S}_t,c\cdot OP{T}_{\delta /4}\right)\right|+c\cdot OP{T}_{\delta /4}}{{\eta }^2{\delta }^2}\right).$$

$$\left|\frac{1}{b}\sum _{i=1}^b\min \left\{\ell \left(\mathbf{\rho },j_i^{(t)}\right),\tau \right\}-{\mathbb{E}}_{j\sim \Gamma }\left[\min \left\{\ell (\mathbf{\rho },j),\tau \right\}\right]\right|\le \tau \cdot \gamma (t,b,\tau ),$$

$$\gamma (t,b,\tau )=\sqrt{\frac{2d\ln \left|\text{textsc}Partition\left(\left\{j_1^{(t)},\dots ,j_b^{(t)}\right\},\tau \right)\right|}{b}}+2\sqrt{\frac{2}{b}\ln \frac{8{\left(\tau bt\right)}^2}{\zeta }}.$$

$$2^{\bar{t}}\le \frac{16\sqrt[4]{1+ϵ}}{\delta }\cdot {\mathbb{E}}_{j\sim \Gamma }\left[\min \left\{\ell \left({\mathbf{\rho }}^{\ast },j\right),t_{\delta /4}\left({\mathbf{\rho }}^{\ast }\right)\right\}\right].$$

$${\mathbb{E}}_{j\sim \Gamma }\left[\min \left\{\ell \left({\mathbf{\rho }}^{\prime },j\right),t_{\delta /2}\left({\mathbf{\rho }}^{\prime }\right)\right\}\right]\le \sqrt{1+ϵ}\cdot {\mathbb{E}}_{j\sim \Gamma }\left[\min \left\{\ell ({\mathbf{\rho }}^{\ast },j),t_{\delta /4}({\mathbf{\rho }}^{\ast })\right\}\right],$$

$${\mathbb{E}}_{j\sim \Gamma }\left[\min \left\{\ell ({\mathbf{\rho }}^{\ast },j),t_{\delta /4}({\mathbf{\rho }}^{\ast })\right\}\right]>\bar{T}.$$

$$\eta =\min \left\{\frac{\sqrt[4]{1+ϵ}-1}{8},\frac{1}{9}\right\}$$

$${\hat{\mathcal{R}}}_{\text{pazocal}S}\left(\text{pazocal}F\right)={\mathbb{E}}_{\mathbf{\sigma }}\left[\underset{f\in \text{pazocal}F}{\sup }\frac{1}{N}\sum _{i=1}^N{\sigma }_i\cdot f\left(x_i\right)\right],$$

$$\left|\frac{1}{N}\sum _{x\in \text{pazocal}S}f(x)-{\mathbb{E}}_{x\sim \text{pazocal}D}[f(x)]\right|\le {\hat{\mathcal{R}}}_{\text{pazocal}S}\left(\text{pazocal}F\right)+2H\sqrt{\frac{2\ln \left(4/\delta \right)}{N}}.$$

$$\frac{1}{\left|\text{pazocal}{S}_t\right|}\sum _{j\in \text{pazocal}{S}_t}{\text{1}}_{\left\{\ell \left(\mathbf{\rho },j\right)\le {\tau }_{\text{pazocal}{P}^{\prime }}\right\}}\ge \frac{\lfloor \left|\text{pazocal}{S}_t\right|(1-3\delta /8)\rfloor }{\left|\text{pazocal}{S}_t\right|}\ge 1-\frac{3\delta }{8}-\frac{1}{\left|\text{pazocal}{S}_t\right|}.$$

$$\underset{j\sim \Gamma }{\mathrm{Pr}}\left[\ell (\mathbf{\rho },j)\le {\tau }_{\text{pazocal}{P}^{\prime }}\right]$$

$$\ge \frac{1}{\left|\text{pazocal}{S}_t\right|}\sum _{j\in \text{pazocal}{S}_t}{\text{1}}_{\left\{\ell \left(\mathbf{\rho },j\right)\le {\tau }_{\text{pazocal}{P}^{\prime }}\right\}}-\sqrt{\frac{2d\ln \left|\text{textsc}Partition\left(\text{pazocal}{S}_t,2^t\right)\right|}{\left|\text{pazocal}{S}_t\right|}}-2\sqrt{\frac{2}{\left|\text{pazocal}{S}_t\right|}\ln \frac{8{\left(2^t\left|\text{pazocal}{S}_t\right|t\right)}^2}{\zeta }},$$

$$\underset{j\sim \Gamma }{\mathrm{Pr}}\left[\ell (\mathbf{\rho },j)\le {\tau }_{\text{pazocal}{P}^{\prime }}\right]\ge \frac{1}{\left|\text{pazocal}{S}_t\right|}\sum _{j\in \text{pazocal}{S}_t}{\text{1}}_{\left\{\ell \left(\mathbf{\rho },j\right)\le {\tau }_{\text{pazocal}{P}^{\prime }}\right\}}-\eta \delta .$$

$$\underset{j\sim \Gamma }{\mathrm{Pr}}\left[\ell (\mathbf{\rho },j)>{\tau }_{\text{pazocal}{P}^{\prime }}\right]=\underset{j\sim \Gamma }{\mathrm{Pr}}\left[\ell (\mathbf{\rho },j)\ge {\tau }_{\text{pazocal}{P}^{\prime }}+1\right]<\frac{\delta }{2}.$$

$$1-\frac{1}{\left|\text{pazocal}{S}_t\right|}\sum _{j\in \text{pazocal}{S}_t}{\text{1}}_{\left\{\ell \left(\mathbf{\rho },j\right)\le {\tau }_{\text{pazocal}{P}^{\prime }}-1\right\}}$$

$$=\frac{1}{\left|\text{pazocal}{S}_t\right|}\sum _{j\in \text{pazocal}{S}_t}\left(1-{\text{1}}_{\left\{\ell \left(\mathbf{\rho },j\right)<{\tau }_{\text{pazocal}{P}^{\prime }}\right\}}\right)$$

$$=\frac{1}{\left|\text{pazocal}{S}_t\right|}\sum _{j\in \text{pazocal}{S}_t}{\text{1}}_{\left\{\ell \left(\mathbf{\rho },j\right)\ge {\tau }_{\text{pazocal}{P}^{\prime }}\right\}}$$

$$\ge \frac{\left|\text{pazocal}{S}_t\right|-\lfloor \left|\text{pazocal}{S}_t\right|(1-3\delta /8)\rfloor }{\left|\text{pazocal}{S}_t\right|}$$

$$\ge \frac{3\delta }{8}.$$

$$\underset{j\sim \Gamma }{\mathrm{Pr}}\left[\ell (\mathbf{\rho },j)\le {\tau }_{\text{pazocal}{P}^{\prime }}-1\right]$$

$$\le$$

$$<$$

$$\underset{j\sim \Gamma }{\mathrm{Pr}}\left[\ell (\mathbf{\rho },j)\le {\tau }_{\text{pazocal}{P}^{\prime }}-1\right]<1-\frac{3\delta }{8}+\eta \delta .$$

$$t_{\delta /4}(\mathbf{\rho })={\text{argmax}}_{\tau \in \mathbb{Z}}\left\{\underset{j\sim \Gamma }{\mathrm{Pr}}[\ell (\mathbf{\rho },j)\ge \tau ]\ge \delta /4\right\},$$

$$\underset{j\sim \Gamma }{\mathrm{Pr}}\left[\ell (\mathbf{\rho },j)\le 2^{\bar{t}-1}\right]$$

$$\le$$

Full text

1 Introduction

Similar combinatorial problems often arise in seemingly unrelated disciplines. Integer programs, for example, model problems in fields ranging from computational biology to economics. To facilitate customization, algorithms often come with tunable parameters that significantly impact the computational resources they consume, such as runtime. Hand-tuning parameters can be time consuming and may lead to sub-optimal results. In this work, we develop the foundations of automated algorithm configuration via machine learning. A key challenge we face is that in order to evaluate a configuration’s requisite computational resources, the learning algorithm itself must expend those resources.

To frame algorithm configuration as a machine learning problem, we assume sample access to an unknown distribution over problem instances, such as the integer programs an airline solves day to day. The learning algorithm uses samples to determine parameters that, ideally, will have strong expected performance. Researchers have studied this configuration model for decades, leading to advances in artificial intelligence [43], computational biology [19], and many other fields. This approach has also been used in industry for tens of billions of dollars of combinatorial auctions [37].

Recently, two lines of research have emerged that explore the theoretical underpinnings of algorithm configuration. One provides sample complexity guarantees, bounding the number of samples sufficient to ensure that an algorithm’s performance on average over the samples generalizes to its expected performance on the distribution [22, 5, 6]. These sample complexity bounds apply no matter how the learning algorithm operates, and these papers do not include learning algorithms that extend beyond exhaustive search.

The second line of research provides algorithms for finding nearly-optimal configurations from a finite set [29, 30, 39, 40]. These algorithms can also be used when the parameter space is infinite: for any $\gamma\in(0,1)$, first sample $\tilde{\Omega}(1/\gamma)$ configurations, and then run the algorithm over this finite set. The authors guarantee that the output configuration will be within the top $\gamma$-quantile. If there is only a small region of high-performing parameters, however, the uniform sample might completely miss all good parameters. Algorithm configuration problems with only tiny pockets of high-performing parameters do indeed exist: Balcan et al. [6] present distributions over integer programs where the optimal parameters lie within an arbitrarily small region of the parameter space. For any parameter within that region, branch-and-bound—the most widely-used integer programming algorithm—terminates instantaneously. Using any other parameter, branch-and-bound takes an exponential number of steps. This region of optimal parameters can be made so small that any random sampling technique would require an arbitrarily large sample of parameters to hit that region. We discuss this example in more detail in Section 5.

This paper marries these two lines of research. We present an algorithm that identifies a finite set of promising parameters within an infinite set, given sample access to a distribution over problem instances. We prove that this set contains a nearly optimal parameter with high probability. The set can serve as the input to a configuration algorithm for finite parameter spaces [29, 30, 39, 40], which we prove will then return a nearly optimal parameter from the infinite set.

An obstacle in our approach is that the loss function measuring an algorithm’s performance as a function of its parameters often exhibits jump discontinuities: a nudge to the parameters can trigger substantial changes in the algorithm’s behavior. In order to provide guarantees, we must tease out useful structure in the configuration problems we study.

The structure we identify is simple yet ubiquitous in combinatorial domains: our approach applies to any configuration problem where the algorithm’s performance as a function of its parameters is piecewise constant. Prior research has demonstrated that algorithm configuration problems from diverse domains exhibit this structure. For example, Balcan et al. [6] uncovered this structure for branch-and-bound algorithm configuration. Many corporations must regularly solve reams of integer programs, and therefore require highly customized solvers. For example, integer programs are a part of many mesh processing pipelines in computer graphics [14]. Animation studios with thousands of meshes require carefully tuned solvers which, thus far, domain experts have handcrafted [15]. Our algorithm can be used to find configurations that minimize the branch-and-bound tree size. Balcan et al. [5] also exhibit this piecewise-constant structure in the context of linkage-based hierarchical clustering algorithms. The algorithm families they study interpolate between the classic single-, complete-, and average-linkage procedures. Building the cluster hierarchy is expensive: the best-known algorithm’s runtime is $\tilde{O}(n^{2})$ given $n$ datapoints  [34]. As with branch-and-bound, our algorithm finds configurations that return satisfactory clusterings while minimizing the hierarchy tree size.

We now describe our algorithm at a high level. Let $\ell$ be a loss function where $\ell(\bm{\rho},j)$ measures the computational resources (running time, for example) required to solve problem instance $j$ using the algorithm parameterized by the vector $\bm{\rho}$. Let $OPT$ be the smallest expected loss111As we describe in Section 2, we compete with a slightly more nuanced benchmark than $OPT$, in line with prior research. $\mathop{\mathbb{E}}_{j\sim\Gamma}[\ell(\bm{\rho},j)]$ of any parameter $\bm{\rho}$, where $\Gamma$ is an unknown distribution over problem instances. Our algorithm maintains upper confidence bound on $OPT$, initially set to $\infty$. On each round $t$, the algorithm begins by drawing a set $\pazocal{S}_{t}$ of sample problem instances. It computes the partition of the parameter space into regions where for each problem instance in $\pazocal{S}_{t}$, the loss $\ell$, capped at $2^{t}$, is a constant function of the parameters. On a given region of this partition, if the average capped loss is sufficiently low, the algorithm chooses an arbitrary parameter from that region and deems it “good.” Once the cap $2^{t}$ has grown sufficiently large compared to the upper confidence bound on $OPT$, the algorithm returns the set of good parameters. We summarize our guarantees informally below.

Theorem 1.1** (Informal).**

The following guarantees hold:

The set of output parameters contains a nearly-optimal parameter with high probability. 2. 2.

Given accuracy parameters $\epsilon$ and $\delta$, the algorithm terminates after $O\left(\ln\left(\sqrt[4]{1+\epsilon}\cdot OPT/\delta\right)\right)$ rounds. 3. 3.

On the algorithm’s final round, let $P$ be the size of the partition the algorithm computes. The number of parameters it outputs is $O\left(P\cdot\ln\left(\sqrt[4]{1+\epsilon}\cdot OPT/\delta\right)\right)$. 4. 4.

The algorithm’s sample complexity on each round $t$ is polynomial in $2^{t}$ (which scales linearly with $OPT$), $\log P$, the parameter space dimension, $\frac{1}{\delta}$, and $\frac{1}{\epsilon}$.

We prove that our sample complexity can be exponentially better than the best-known uniform convergence bound. Moreover, it can find strong configurations in scenarios where uniformly sampling configurations will fail.

2 Problem definition

The algorithm configuration model we adopt is a generalization of the model from prior research [29, 30, 39, 40]. There is a set $\Pi$ of problem instances and an unknown distribution $\Gamma$ over $\Pi$. For example, this distribution might represent the integer programs an airline solves day to day. Each algorithm is parameterized by a vector $\bm{\rho}\in\pazocal{P}\subseteq\mathbb{R}^{d}$. At a high level, we assume we can set a budget on the computational resources the algorithm consumes, which we quantify using an integer $\tau\in\mathbb{Z}_{\geq 0}$. For example, $\tau$ might measure the maximum running time we allow the algorithm. There is a utility function $u:\pazocal{P}\times\Pi\times\mathbb{Z}_{\geq 0}\to\{0,1\}$, where $u(\bm{\rho},j,\tau)=1$ if and only if the algorithm parameterized by $\bm{\rho}$ returns a solution to the instance $j$ given a budget of $\tau$. We make the natural assumption that the algorithm is more likely to find a solution the higher its budget: $u(\bm{\rho},j,\tau)\geq u(\bm{\rho},j,\tau^{\prime})$ for $\tau\geq\tau^{\prime}$. Finally, there is a loss function $\ell:\pazocal{P}\times\Pi\to\mathbb{Z}_{\geq 0}$ which measures the minimum budget the algorithm requires to find a solution. Specifically, $\ell(\bm{\rho},j)=\infty$ if $u(\bm{\rho},j,\tau)=0$ for all $\tau$, and otherwise, $\ell(\bm{\rho},j)=\textnormal{argmin}\left\{\tau:u(\bm{\rho},j,\tau)=1\right\}$. In Section 2.1, we provide several examples of this problem definition instantiated for combinatorial problems.

The distribution $\Gamma$ over problem instances is unknown, so we use samples from $\Gamma$ to find a parameter vector $\hat{\bm{\rho}}\in\pazocal{P}$ with small expected loss. Ideally, we could guarantee that

[TABLE]

Unfortunately, this ideal goal is impossible to achieve with a finite number of samples, even in the extremely simple case where there are only two configurations, as illustrated below.

Example 2.1**.**

[Weisz et al. [40]] Let $\pazocal{P}=\{1,2\}$ be a set of two configurations. Suppose that the loss of the first configuration is 2 for all problem instances: $\ell(1,j)=2$ for all $j\in\Pi$. Meanwhile, suppose that $\ell(2,j)=\infty$ with probability $\delta$ for some $\delta\in(0,1)$ and $\ell(2,j)=1$ with probability $1-\delta$. In this case, $\mathop{\mathbb{E}}_{j\sim\Gamma}[\ell(1,j)]=2$ and $\mathop{\mathbb{E}}_{j\sim\Gamma}[\ell(2,j)]=\infty$. In order for any algorithm to verify that the first configuration’s expected loss is substantially better than the second’s, it must sample at least one problem instance $j$ such that $\ell(2,j)=\infty$. Therefore, it must sample $\Omega(1/\delta)$ problem instances, a lower bound that approaches infinity as $\delta$ shrinks. As a result, it is impossible to give a finite bound on the number of samples sufficient to find a parameter $\hat{\bm{\rho}}$ that satisfies Equation (1).

The obstacle that this example exposes is that some configurations might have an enormous loss on a few rare problem instances. To deal with this impossibility result, Weisz et al. [39, 40], building off of work by Kleinberg et al. [29], propose a relaxed notion of approximate optimality. To describe this relaxation, we introduce the following notation. Given $\delta\in(0,1)$ and a parameter vector $\bm{\rho}\in\pazocal{P}$, let $t_{\delta}(\bm{\rho})$ be the largest cutoff $\tau\in\mathbb{Z}_{\geq 0}$ such that the probability $\ell(\bm{\rho},j)$ is greater than $\tau$ is at least $\delta$. Mathematically, $t_{\delta}(\bm{\rho})=\textnormal{argmax}_{\tau\in\mathbb{Z}}\left\{\Pr_{j\sim\Gamma}[\ell(\bm{\rho},j)\geq\tau]\geq\delta\right\}$. The value $t_{\delta}(\bm{\rho})$ can be thought of as the beginning of the loss function’s “$\delta$-tail.”

We illustrate the definition of $t_{\delta}(\bm{\rho})$ in Figure 1. We now define the relaxed notion of approximate optimality by Weisz et al. [39].

Definition 2.2** ($\left(\epsilon,\delta,\pazocal{P}\right)$-optimality).**

A parameter vector $\hat{\bm{\rho}}$ is $\left(\epsilon,\delta,\pazocal{P}\right)$-optimal if

[TABLE]

In other words, a parameter vector $\hat{\bm{\rho}}$ is $\left(\epsilon,\delta,\pazocal{P}\right)$-optimal if its $\delta$-capped expected loss is within a $(1+\epsilon)$-factor of the optimal $\delta/2$-capped expected loss.222The fraction $\delta/2$ can be replaced with any $c\delta$ for $c\in(0,1)$. Ideally, we would replace $\delta/2$ with $\delta$, but the resulting property would be impossible to verify with high probability [39]. To condense notation, we write $OPT_{c\delta}:=\inf_{\bm{\rho}\in\pazocal{P}}\left\{\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell\left(\bm{\rho},j\right),t_{c\delta}(\bm{\rho})\right\}\right]\right\}$. If an algorithm returns an $\left(\epsilon,\delta,\bar{\pazocal{P}}\right)$-optimal parameter from within a finite set $\bar{\pazocal{P}}$, we call it a configuration algorithm for finite parameter spaces. Weisz et al. [40] provide one such algorithm, CapsAndRuns.

2.1 Example applications

In this section, we provide several instantiations of our problem definition in combinatorial domains.

Tree search.

Tree search algorithms, such as branch-and-bound, are the most widely-used tools for solving combinatorial problems, such as (mixed) integer programs and constraint satisfaction problems. These algorithms recursively partition the search space to find an optimal solution, organizing this partition as a tree. Commercial solvers such as CPLEX, which use tree search under the hood, come with hundreds of tunable parameters. Researchers have developed machine learning algorithms for tuning these parameters [24, 20, 23, 4, 27, 28, 32, 42, 6]. Given parameters $\bm{\rho}$ and a problem instance $j$, we might define the budget $\tau$ to cap the size of the tree the algorithm builds. In that case, the utility function is defined such that $u(\bm{\rho},j,\tau)=1$ if and only if the algorithm terminates, having found the optimal solution, after building a tree of size $\tau$. The loss $\ell(\bm{\rho},j)$ equals the size of the entire tree built by the algorithm parameterized by $\bm{\rho}$ given the instance $j$ as input.

Clustering.

Given a set of datapoints and the distances between each point, the goal in clustering is to partition the points into subsets so that points within any set are “similar.” Clustering algorithms are used to group proteins by function, classify images by subject, and myriad other applications. Typically, the quality of a clustering is measured by an objective function, such as the classic $k$-means, $k$-median, or $k$-center objectives. Unfortunately, it is NP-hard to determine the clustering that minimizes any of these objectives. As a result, researchers have developed a wealth of approximation and heuristic clustering algorithms. However, no one algorithm is optimal across all applications.

Balcan et al. [5] provide sample complexity guarantees for clustering algorithm configuration. Each problem instance is a set of datapoints and there is a distribution over clustering problem instances. They analyze several infinite classes of clustering algorithms. Each of these algorithms begins with a linkage-based step and concludes with a dynamic programming step. The linkage-based routine constructs a hierarchical tree of clusters. At the beginning of the process, each datapoint is in a cluster of its own. The algorithm sequentially merges the clusters into larger clusters until all elements are in the same cluster. There are many ways to build this tree: merge the clusters that are closest in terms of their two closest points (single-linkage), their two farthest points (complete-linkage), or on average over all pairs of points (average-linkage). These linkage procedures are commonly used in practice [3, 36, 41] and come with theoretical guarantees. Balcan et al. [5] study an infinite parameterization, $\rho$-linkage, that interpolates between single-, average-, and complete-linkage. After building the cluster tree, the dynamic programming step returns the pruning of this tree that minimizes a fixed objective function, such as the $k$-means, $k$-median, or $k$-center objectives.

Building the full hierarchy is expensive because the best-known algorithm’s runtime is $O(n^{2}\log n)$, where $n$ is the number of datapoints  [34]. It is not always necessary, however, to build the entire tree: the algorithm can preemptively terminate the linkage step after $\tau$ merges, then use dynamic programming to recover the best pruning of the cluster forest. We refer to this variation as $\tau$-capped $\rho$-linkage. To evaluate the resulting clustering, we assume there is a cost function $c:\pazocal{P}\times\Pi\times\mathbb{Z}\to\mathbb{R}$ where $c(\rho,j,\tau)$ measures the quality of the clustering $\tau$-capped $\rho$-linkage returns, given the instance $j$ as input. We assume there is a threshold $\theta_{j}$ where the clustering is admissible if and only if $c(\rho,j,\tau)\leq\theta_{j}$, which means the utility function is defined as $u(\rho,j,\tau)=\textbf{1}_{\left\{c(\rho,j,\tau)\leq\theta_{j}\right\}}$. For example, $c(\rho,j,\tau)$ might measure the clustering’s $k$-means objective value, and $\theta_{j}$ might equal the optimal $k$-means objective value (obtained only for the training instances via an expensive computation) plus an error term.

3 Data-dependent discretizations of infinite parameter spaces

We begin this section by proving an intuitive fact: given a finite subset $\bar{\pazocal{P}}\subset\pazocal{P}$ of parameters that contains at least one “sufficiently good” parameter, a configuration algorithm for finite parameter spaces, such as CapsAndRuns [40], returns a parameter that’s nearly optimal over the infinite set $\pazocal{P}$. Therefore, our goal is to provide an algorithm that takes as input an infinite parameter space and returns a finite subset that contains at least one good parameter. A bit more formally, a parameter is “sufficiently good” if its $\delta/2$-capped expected loss is within a $\sqrt{1+\epsilon}$-factor of $OPT_{\delta/4}$. We say a finite parameter set $\bar{\pazocal{P}}$ is an $\left(\epsilon,\delta\right)$-optimal subset if it contains a good parameter.

Definition 3.1** ($\left(\epsilon,\delta\right)$-optimal subset).**

A finite set $\bar{\pazocal{P}}\subset\pazocal{P}$ is an $\left(\epsilon,\delta\right)$-optimal subset if there is a vector $\hat{\bm{\rho}}\in\bar{\pazocal{P}}$ such that $\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell\left(\hat{\bm{\rho}},j\right),t_{\delta/2}\left(\hat{\bm{\rho}}\right)\right\}\right]\leq\sqrt{1+\epsilon}\cdot OPT_{\delta/4}.$

We now prove that given an $(\epsilon,\delta)$-optimal subset $\bar{\pazocal{P}}\subset\pazocal{P}$, a configuration algorithm for finite parameter spaces returns a nearly optimal parameter from the infinite space $\pazocal{P}$.

Theorem 3.2**.**

Let $\bar{\pazocal{P}}\subset\pazocal{P}$ be an $(\epsilon,\delta)$-optimal subset and let $\epsilon^{\prime}=\sqrt{1+\epsilon}-1$. Suppose $\hat{\bm{\rho}}\in\bar{\pazocal{P}}$ is $\left(\epsilon^{\prime},\delta,\bar{\pazocal{P}}\right)$-optimal. Then $\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell\left(\hat{\bm{\rho}},j\right),t_{\delta}\left(\hat{\bm{\rho}}\right)\right\}\right]\leq(1+\epsilon)\cdot OPT_{\delta/4}$.

Proof.

Since the parameter $\hat{\bm{\rho}}$ is $\left(\epsilon^{\prime},\delta,\bar{\pazocal{P}}\right)$-optimal, we know that $\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell\left(\hat{\bm{\rho}},j\right),t_{\delta}\left(\hat{\bm{\rho}}\right)\right\}\right]\leq\sqrt{1+\epsilon}\cdot\min_{\bm{\rho}\in\bar{\pazocal{P}}}\left\{\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell\left(\bm{\rho},j\right),t_{\delta/2}(\bm{\rho})\right\}\right]\right\}$. (We use a minimum instead of an infimum because $\bar{\pazocal{P}}$ is a finite set by Definition 3.1.) The set $\bar{\pazocal{P}}$ is an $(\epsilon,\delta)$-optimal subset of the parameter space $\pazocal{P}$, so there exists a parameter vector $\bm{\rho}^{\prime}\in\bar{\pazocal{P}}$ such that $\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell\left(\bm{\rho}^{\prime},j\right),t_{\delta/2}\left(\bm{\rho}^{\prime}\right)\right\}\right]\leq\sqrt{1+\epsilon}\cdot OPT_{\delta/4}$. Therefore,

[TABLE]

so the theorem statement holds. ∎

4 Our main result: Algorithm for learning $(\epsilon,\delta)$-optimal subsets

We present an algorithm for learning $(\epsilon,\delta)$-optimal subsets for configuration problems that satisfy a simple, yet ubiquitous structure: for any problem instance $j$, the loss function $\ell(\cdot,j)$ is piecewise constant. This structure has been observed throughout a diverse array of configuration problems ranging from clustering to integer programming [5, 6]. More formally, this structure holds if for any problem instance $j\in\Pi$ and cap $\tau\in\mathbb{Z}_{\geq 0}$, there is a finite partition of the parameter space $\pazocal{P}$ such that in any one region $R$ of this partition, for all pairs of parameter vectors $\bm{\rho},\bm{\rho}^{\prime}\in R$, $\min\left\{\ell(\bm{\rho},j),\tau\right\}=\min\left\{\ell(\bm{\rho}^{\prime},j),\tau\right\}$.

To exploit this piecewise-constant structure, we require access to a function Partition that takes as input a set $\pazocal{S}$ of problem instances and an integer $\tau$ and returns this partition of the parameters. Namely, it returns a set of tuples $\left(\pazocal{P}_{1},z_{1},\bm{\tau}_{1}\right),\dots,\left(\pazocal{P}_{k},z_{k},\bm{\tau}_{k}\right)\in 2^{\pazocal{P}}\times[0,1]\times\mathbb{Z}^{|\pazocal{S}|}$ such that:

The sets $\pazocal{P}_{1},\dots,\pazocal{P}_{k}$ make up a partition of $\pazocal{P}$. 2. 2.

For all subsets $\pazocal{P}_{i}$ and vectors $\bm{\rho},\bm{\bm{\rho}^{\prime}}\in\pazocal{P}_{i}$, $\frac{1}{|\pazocal{S}|}\sum_{j\in\pazocal{S}}\textbf{1}_{\left\{\ell(\bm{\rho},j)\leq\tau\right\}}=\frac{1}{|\pazocal{S}|}\sum_{j\in\pazocal{S}}\textbf{1}_{\left\{\ell(\bm{\rho}^{\prime},j)\leq\tau\right\}}=z_{i}.$ 3. 3.

For all subsets $\pazocal{P}_{i}$, all $\bm{\rho},\bm{\bm{\rho}^{\prime}}\in\pazocal{P}_{i}$, and all $j\in\pazocal{S}$, $\min\left\{\ell(\bm{\rho},j),\tau\right\}=\min\left\{\ell(\bm{\rho}^{\prime},j),\tau\right\}=\tau_{i}[j].$

We assume the number of tuples Partition returns is monotone: if $\tau\leq\tau^{\prime}$, then $\left|\textsc{Partition}(\pazocal{S},\tau)\right|\leq\left|\textsc{Partition}(\pazocal{S},\tau^{\prime})\right|$ and if $\pazocal{S}\subseteq\pazocal{S}^{\prime}$, then $\left|\textsc{Partition}(\pazocal{S},\tau)\right|\leq\left|\textsc{Partition}(\pazocal{S}^{\prime},\tau)\right|$.

As we describe in Appendix C, results from prior research imply guidance for implementing Partition in the contexts of clustering and integer programming. For example, in the clustering application we describe in Section 2.1, the distribution $\Gamma$ is over clustering instances. Suppose $n$ is an upper bound on the number of points in each instance. Balcan et al. [5] prove that for any set $\pazocal{S}$ of samples and any cap $\tau$, in the worst case, $\left|\textsc{Partition}(\pazocal{S},\tau)\right|=O\left(|\pazocal{S}|n^{8}\right)$, though empirically, $\left|\textsc{Partition}(\pazocal{S},\tau)\right|$ is often several orders of magnitude smaller [9]. Balcan et al. [5] and Balcan et al. [9] provide guidance for implementing Partition.

High-level description of algorithm.

We now describe our algorithm for learning $(\epsilon,\delta)$-optimal subsets. See Algorithm 1 for the pseudocode. The algorithm maintains a variable $T$, initially set to $\infty$, which roughly represents an upper confidence bound on $OPT_{\delta/4}$. It also maintains a set $\pazocal{G}$ of parameters which the algorithm believes might be nearly optimal. The algorithm begins by aggressively capping the maximum loss $\ell$ it computes by 1. At the beginning of each round, the algorithm doubles this cap until the cap grows sufficiently large compared to the upper confidence bound $T$. At that point, the algorithm terminates. On each round $t$, the algorithm draws a set $\pazocal{S}_{t}$ of samples (Step 5) that is just large enough to estimate the expected $2^{t}$-capped loss $\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell(\bm{\rho},j),2^{t}\right\}\right]$ for every parameter $\bm{\rho}\in\pazocal{P}$. The number of samples it draws is a data-dependent quantity that depends on empirical Rademacher complexity [31, 11].

Next, the algorithm evaluates the function $\textsc{Partition}\left(\pazocal{S}_{t},2^{t}\right)$ to obtain the tuples

[TABLE]

By definition of this function, for all subsets $\pazocal{P}_{i}$ and parameter vector pairs $\bm{\rho},\bm{\rho}^{\prime}\in\pazocal{P}_{i}$, the fraction of instances $j\in\pazocal{S}_{t}$ with $\ell(\bm{\rho},j)\leq 2^{t}$ is equal to the fraction of instances $j\in\pazocal{S}_{t}$ with $\ell(\bm{\rho}^{\prime},j)\leq 2^{t}$. In other words, $\frac{1}{|\pazocal{S}_{t}|}\sum_{j\in\pazocal{S}_{t}}\textbf{1}_{\left\{\ell(\bm{\rho},j)\leq 2^{t}\right\}}=\frac{1}{|\pazocal{S}_{t}|}\sum_{j\in\pazocal{S}_{t}}\textbf{1}_{\left\{\ell(\bm{\rho}^{\prime},j)\leq 2^{t}\right\}}=z_{i}.$ If this fraction is sufficiently high (at least $1-3\delta/8$), the algorithm adds $\pazocal{P}_{i}$ to the set of good parameters $\pazocal{G}$ (Step 8). The algorithm estimates the $\delta/4$-capped expected loss of the parameters contained $\pazocal{P}_{i}$, and if this estimate is smaller than the current upper confidence bound $T$ on $OPT_{\delta/4}$, it updates $T$ accordingly (Steps 9 through 11). Once the cap $2^{t}$ has grown sufficiently large compared to the upper confidence bound $T$, the algorithm returns an arbitrary parmeter from each set in $\pazocal{G}$.

Algorithm analysis.

We now provide guarantees on Algorithm 1’s performance. We denote the values of $t$ and $T$ at termination by $\bar{t}$ and $\bar{T}$, and we denote the state of the set $\pazocal{G}$ at termination by $\bar{\pazocal{G}}$. For each set $\pazocal{P}^{\prime}\in\bar{\pazocal{G}}$, we use the notation $\tau_{\pazocal{P}^{\prime}}$ to denote the value $\tau_{\lfloor|\pazocal{S}_{t}|\left(1-3\delta/8\right)\rfloor}$ in Step 10 during the iteration $t$ that $\pazocal{P}^{\prime}$ is added to $\pazocal{G}$.

Theorem 4.1**.**

With probability $1-\zeta$, the following conditions hold, with

[TABLE]

Algorithm 1 terminates after $\bar{t}=O\left(\log\left(c\cdot OPT_{\delta/4}\right)\right)$ iterations. 2. 2.

Algorithm 1 returns an $(\epsilon,\delta)$-optimal set of parameters of size at most

[TABLE] 3. 3.

The sample complexity on round $t\in\left[\bar{t}\right]$, $\left|\pazocal{S}_{t}\right|$, is

[TABLE]

Proof.

We split the proof into separate lemmas. Lemma 4.4 proves Part 1. Lemma 4.6 as well as Lemma B.11 in Appendix B prove Part 2. Finally, Part 3 follows from classic results in learning theory on Rademacher complexity. In particular, it follows from an inversion of the inequality in Step 5 and the fact that $2^{t}\leq 2^{\bar{t}}\leq c\cdot OPT_{\delta/4}$, as we prove in Lemma 4.4. ∎

Theorem 4.1 hinges on the assumption that the samples $\pazocal{S}_{1},\dots,\pazocal{S}_{\bar{t}}$ Algorithm 1 draws in Step 5 are sufficiently representative of the distribution $\Gamma$, formalized as follows:

Definition 4.2** ($\zeta$-representative run).**

For each round $t\in\left[\bar{t}\right]$, denote the samples in $\pazocal{S}_{t}$ as $\pazocal{S}_{t}=\left\{j_{i}^{(t)}:i\in\left[\left|\pazocal{S}_{t}\right|\right]\right\}$. We say that Algorithm 1 has a $\zeta$-representative run if for all rounds $t\in\left[\bar{t}\right]$, all integers $b\in\left[\left|\pazocal{S}_{t}\right|\right]$, all caps $\tau\in\mathbb{Z}_{\geq 0}$, and all parameters $\bm{\rho}\in\pazocal{P}$, the following conditions hold:

The average number of instances $j_{i}^{(t)}\in\left\{j_{1}^{(t)},\dots,j_{b}^{(t)}\right\}$ with loss smaller than $\tau$ nearly matches the probability that $\ell(\bm{\rho},j)\leq\tau$: $\left|\frac{1}{b}\sum_{i=1}^{b}\textbf{1}_{\left\{\ell\left(\bm{\rho},j_{i}^{(t)}\right)\leq\tau\right\}}-\Pr_{j\sim\Gamma}\left[\ell(\bm{\rho},j)\leq\tau\right]\right|\leq\gamma(t,b,\tau)$, and 2. 2.

The average $\tau$-capped loss of the instances $j_{1}^{(t)},\dots,j_{b}^{(t)}$ nearly matches the expected $\tau$-capped loss:

[TABLE]

where

[TABLE]

In Step 5, we ensure $\pazocal{S}_{t}$ is large enough that Algorithm 1 has a $\zeta$-representative run with probability $1-\zeta$.

Lemma 4.3**.**

With probability $1-\zeta$, Algorithm 1 has a $\zeta$-representative run.

Lemma 4.3 is a corollary of a Rademacher complexity analysis which we include in Appendix B (Lemma B.1). Intuitively, there are only $\left|\textsc{Partition}(\pazocal{S},\tau)\right|$ algorithms with varying $\tau$-capped losses over any set of samples $\pazocal{S}$. We can therefore invoke Massart’s finite lemma [35], which guarantees that each set $\pazocal{S}_{t}$ is sufficiently large to ensure that Algorithm 1 indeed has a $\zeta$-representative run. The remainder of our analysis will assume that Algorithm 1 has a $\zeta$-representative run.

Number of iterations until termination.

We begin with a proof sketch of the first part of Theorem 4.1. The full proof is in Appendix B.

Lemma 4.4**.**

Suppose Algorithm 1 has a $\zeta$-representative run. Then $2^{\bar{t}}\leq\frac{16}{\delta}\sqrt[4]{1+\epsilon}\cdot OPT_{\delta/4}$.

Proof sketch.

By definition of $OPT_{\delta/4}$, for every $\gamma>0$, there exists a vector $\bm{\rho}^{*}\in\pazocal{P}$ whose $\delta/4$-capped expected loss is within a $\gamma$-factor of optimal: $\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell\left(\bm{\rho}^{*},j\right),t_{\delta/4}\left(\bm{\rho}^{*}\right)\right\}\right]\leq OPT_{\delta/4}+\gamma.$ We prove this vector’s $\delta/4$-capped expected loss bounds $\bar{t}$:

[TABLE]

This implies the lemma statement holds. We split the proof of Inequality (2) into two cases: one where the vector $\bm{\rho}^{*}$ is contained within a set $\pazocal{P}^{\prime}\in\bar{\pazocal{G}}$, and the other where it is not. In the latter case, Lemma 4.5 bounds $2^{\bar{t}}$ by $\frac{8}{\delta}\cdot\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell\left(\bm{\rho}^{*},j\right),t_{\delta/4}\left(\bm{\rho}^{*}\right)\right\}\right]$, which implies Inequality (2) holds. We leave the other case to the full proof in Appendix B. ∎

In the next lemma, we prove the upper bound on $2^{\bar{t}}$ that we use in Lemma 4.4. The full proof is in Appendix B.

Lemma 4.5**.**

Suppose Algorithm 1 has a $\zeta$-representative run. For any parameter vector $\bm{\rho}\not\in\bigcup_{\pazocal{P}^{\prime}\in\bar{\pazocal{G}}}\pazocal{P}^{\prime}$, $2^{\bar{t}}\leq\frac{8}{\delta}\cdot\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell(\bm{\rho},j),t_{\delta/4}(\bm{\rho})\right\}\right].$

Proof sketch.

The last round that Algorithm 1 adds any subset to the set $\pazocal{G}$ is round $\bar{t}-1$. For ease of notation, let $\bar{\pazocal{S}}=\pazocal{S}_{\bar{t}-1}$. Since $\bm{\rho}$ is not an element of any set in $\pazocal{G}$, the cap $2^{\bar{t}-1}$ must be too small compared to the average loss of the parameter $\bm{\rho}$. Specifically, it must be that $\frac{1}{\left|\bar{\pazocal{S}}\right|}\sum_{j\in\bar{\pazocal{S}}}\textbf{1}_{\left\{\ell\left(\bm{\rho},j\right)\leq 2^{\bar{t}-1}\right\}}<1-\frac{3\delta}{8}$. Since Algorithm 1 had a $\zeta$-representative run, the probability the loss of $\bm{\rho}$ is smaller than $2^{\bar{t}-1}$ converges to the fraction of samples with loss smaller than $2^{\bar{t}-1}$: $\Pr_{j\sim\Gamma}\left[\ell(\bm{\rho},j)\leq 2^{\bar{t}-1}\right]\leq\frac{1}{\left|\bar{\pazocal{S}}\right|}\sum_{j\in\bar{\pazocal{S}}}\textbf{1}_{\left\{\ell\left(\bm{\rho},j\right)\leq 2^{\bar{t}-1}\right\}}+\eta\delta,$ so $\Pr_{j\sim\Gamma}\left[\ell(\bm{\rho},j)\leq 2^{\bar{t}-1}\right]<1-(3/8-\eta)\delta$. Since $\eta\leq 1/9$, it must be that $\Pr_{j\sim\Gamma}\left[\ell(\bm{\rho},j)\geq 2^{\bar{t}-1}\right]\geq\delta/4$, so by definition of $t_{\delta/4}(\bm{\rho})$, we have that $2^{\bar{t}-1}\leq t_{\delta/4}(\bm{\rho})$. Therefore, $\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell(\bm{\rho},j),t_{\delta/4}(\bm{\rho})\right\}\right]\geq\frac{\delta}{4}\cdot t_{\delta/4}(\bm{\rho})\geq 2^{{\bar{t}}-3}\delta$. ∎

Optimality of Algorithm 1’s output.

Next, we provide a proof sketch of the second part of Theorem 4.1, which guarantees that Algorithm 1 returns an $(\epsilon,\delta)$-optimal subset. The full proof is in Appendix B. For each set $\pazocal{P}^{\prime}\in\bar{\pazocal{G}}$, $\tau_{\pazocal{P}^{\prime}}$ denotes the value of $\tau_{\lfloor|\pazocal{S}_{t}|\left(1-3\delta/8\right)\rfloor}$ in Step 10 of Algorithm 1 during the iteration $t$ that $\pazocal{P}^{\prime}$ is added to $\pazocal{G}$.

Lemma 4.6**.**

If Algorithm 1 has a $\zeta$-representative run, it returns an $(\epsilon,\delta)$-optimal subset.

Proof sketch.

By definition of $OPT_{\delta/4}$, for every $\gamma>0$, there is a vector $\bm{\rho}^{*}\in\pazocal{P}$ such that $\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell(\bm{\rho}^{*},j),t_{\delta/4}(\bm{\rho}^{*})\right\}\right]\leq OPT_{\delta/4}+\gamma.$ Let $\pazocal{P}^{*}$ be the output of Algorithm 1. We claim there exists a parameter $\bm{\rho}^{\prime}\in\pazocal{P}^{*}$ such that

[TABLE]

which implies the lemma statement. There are two cases: either the vector $\bm{\rho}^{*}$ is contained within a set $\pazocal{P}^{\prime}\in\bar{\pazocal{G}}$, or it is not. In this sketch, we analyze the latter case.

By Lemma 4.5, we know that $\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell\left(\bm{\rho}^{*},j\right),t_{\delta/4}(\bm{\rho}^{*})\right\}\right]\geq 2^{\bar{t}-3}\delta.$ When Algorithm 1 terminates, $2^{\bar{t}-3}\delta$ is greater than the upper confidence bound $\bar{T}$, which means that

[TABLE]

We next derive a lower bound on $\bar{T}$: we prove that there exists a set $\pazocal{P}^{\prime}\in\bar{\pazocal{G}}$ and parameter vector $\bm{\rho}\in\pazocal{P}^{\prime}$ such that $\sqrt[4]{1+\epsilon}\cdot\bar{T}\geq\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell\left(\bm{\rho},j\right),\tau_{\pazocal{P}^{\prime}}\right\}\right]$. Our upper bound on $\bar{T}$ implies that $\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell\left(\bm{\rho},j\right),\tau_{\pazocal{P}^{\prime}}\right\}\right]\leq\sqrt[4]{1+\epsilon}\cdot\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell\left(\bm{\rho}^{*},j\right),t_{\delta/4}(\bm{\rho})\right\}\right]$. Finally, we prove that there is a parameter $\bm{\rho}^{\prime}\in\pazocal{P}^{\prime}\cap\pazocal{P}^{*}$ whose $\delta/2$-capped expected loss is within a $\sqrt[4]{1+\epsilon}$-factor of the expected loss $\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell\left(\bm{\rho},j\right),\tau_{\pazocal{P}^{\prime}}\right\}\right]$. This follows from a proof that $\tau_{\pazocal{P}^{\prime}}$ approximates $t_{\delta/4}(\bm{\rho}^{\prime})$ and the fact that $\bm{\rho}$ and $\bm{\rho^{\prime}}$ are both elements of $\pazocal{P}^{\prime}$. Stringing these inequalities together, we prove that Equation (3) holds. ∎

The second part of Theorem 4.1 also guarantees that the size of the set Algorithm 1 returns is bounded (see Lemma B.11 in Appendix B). Together, Lemmas 4.4 and 4.6, as well as Lemma B.11 in Appendix B, bound the number of iterations Algorithm 1 makes until it returns an $(\epsilon,\delta)$-optimal subset.

5 Comparison to prior research

We now provide comparisons to prior research on algorithm configuration with provable guarantees. Both comparisons revolve around branch-and-bound (B&B) configuration for integer programming (IP), overviewed in Section 2.1.

Uniformly sampling configurations.

Prior research provides algorithms for finding nearly-optimal configurations from a finite set [29, 30, 39, 40]. If the parameter space is infinite and their algorithms optimize over a uniformly-sampled set of $\tilde{\Omega}(1/\gamma)$ configurations, then the output configuration will be within the top $\gamma$-quantile, with high probability. If the set of good parameters is small, however, the uniform sample might not include any of them. Algorithm configuration problems where the high-performing parameters lie within a small region do exist, as we illustrate in the following theorem.

Theorem 5.1** (Balcan et al. [6]).**

For any $\frac{1}{3}<a<b<\frac{1}{2}$ and $n\geq 6$, there are infinitely-many distributions $\Gamma$ over IPs with $n$ variables and a B&B parameter333As we describe in Appendix C.2, $\rho$ controls the variable selection policy. The theorem holds for any node selection policy. with range $[0,1]$ such that:

If $\rho\leq a$, then $\ell(\rho,j)=2^{(n-5)/4}$ with probability $\frac{1}{2}$ and $\ell(\rho,j)=8$ with probability $\frac{1}{2}$. 2. 2.

If $\rho\in(a,b)$, then $\ell(\rho,j)=8$ with probability $1$. 3. 3.

If $\rho\geq b$, then $\ell(\rho,j)=2^{(n-4)/2}$ with probability $\frac{1}{2}$ and $\ell(\rho,j)=8$ with probability $\frac{1}{2}$.

Here, $\ell(\rho,j)$ measures the size of the tree B&B builds using the parameter $\rho$ on the input integer program $j$.

In the above configuration problem, any parameter in the range $(a,b)$ has a loss of 8 with probability 1, which is the minimum possible loss. Any parameter outside of this range has an abysmal expected loss of at least $2^{(n-6)/2}$. In fact, for any $\delta\leq 1/2$, the $\delta$-capped expected loss of any parameter in the range $[0,a]\cup[b,1]$ is at least $2^{(n-6)/2}$. Therefore, if we uniformly sample a finite set of parameters and optimize over this set using an algorithm for finite parameter spaces [29, 30, 39, 40], we must ensure that we sample at least one parameter within $(a,b)$. As $a$ and $b$ converge, however, the required number of samples shoots to infinity, as we formalize below. This section’s omitted proofs are in Appendix C.2.

Theorem 5.2**.**

For the B&B configuration problem in Theorem 5.1, with constant probability over the draw of $m=\left\lfloor 1/(b-a)\right\rfloor$ parameters $\rho_{1},\dots,\rho_{m}\sim\text{Uniform}[0,1]$, $\left\{\rho_{1},\dots,\rho_{m}\right\}\cap(a,b)=\emptyset$.

Meanwhile, Algorithm 1 quickly terminates, having found an optimal parameter, as we describe below.

Theorem 5.3**.**

For the B&B configuration problem in Theorem 5.1, Algorithm 1 terminates after $\tilde{O}(\log 1/\delta)$ iterations, having drawn $\tilde{O}((\delta\eta)^{-2})$ sample problem instances where

[TABLE]

and returns a set containing an optimal parameter in $(a,b)$.

Similarly, Balcan et al. [8] exemplify clustering configuration problems—which we overview in Section 2.1—where the optimal parameters lie within an arbitrarily small region, and any other parameter leads to significantly worse performance. As in Theorem 5.2, this means a uniform sampling of the parameters will fail to find optimal parameters.

Uniform convergence.

Prior research has provided uniform convergence sample complexity bounds for algorithm configuration. These guarantees bound the number of samples sufficient to ensure that for any configuration, its average loss over the samples nearly matches its expected loss.

We prove that in the case of B&B configuration, Algorithm 1 may use far fewer samples to find a nearly optimal configuration than the best-known uniform convergence sample complexity bound. Balcan et al. [6] prove uniform convergence sample complexity guarantees for B&B configuration. They bound the number of samples sufficient to ensure that for any configuration in their infinite parameter space, the size of the search tree B&B builds on average over the samples generalizes to the expected size of the tree it builds. For integer programs over $n$ variables, the best-known sample complexity bound guarantees that $\left(2^{n}/\epsilon^{\prime}\right)^{2}$ samples are sufficient to ensure that the average tree size B&B builds over the samples is within an additive $\epsilon^{\prime}$ factor of the expected tree size [6]. Meanwhile, as we describe in Theorem 5.3, there are B&B configuration problems where our algorithm’s sample complexity bound is significantly better: our algorithm finds an optimal parameter using only $\tilde{O}((\delta\eta)^{-2})$ samples.

6 Conclusion

We presented an algorithm that learns a finite set of promising parameters from an infinite parameter space. It can be used to determine the input to a configuration algorithm for finite parameter spaces, or as a tool for compiling an algorithm portfolio. We proved bounds on the number of iterations before our algorithm terminates, its sample complexity, and the size of its output. A strength of our approach is its modularity: it can determine the input to a configuration algorithm for finite parameter spaces without depending on specifics of that algorithm’s implementation. There is an inevitable tradeoff, however, between modularity and computational efficiency. In future research, our approach can likely be folded into existing configuration algorithms for finite parameter spaces.

Acknowledgments

This material is based on work supported by the National Science Foundation under grants IIS-1718457, IIS-1617590, IIS-1618714, IIS-1901403, CCF-1535967, CCF-1910321, SES-1919453, and CCF-1733556, the ARO under award W911NF-17-1-0082, an Amazon Research Award, an AWS Machine Learning Research Award, a Bloomberg Data Science research grant, a fellowship from Carnegie Mellon University’s Center for Machine Learning and Health, and the IBM PhD Fellowship.

Appendix A Review of Rademacher complexity

At a high level, Rademacher complexity measures the extent to which a class of functions fit random noise. Intuitively, this measures the richness of a function class because more complex classes should be able to fit random noise better than simple classes. Empirical Rademacher complexity can be measured on the set of samples and implies generalization guarantees that improve based on structure exhibited by the set of samples. Formally, let $\pazocal{F}\subseteq[0,H]^{\pazocal{X}}$ be an abstract function class mapping elements of a domain $\pazocal{X}$ to the interval $[0,H]$. Given a set $\pazocal{S}=\left\{x_{1},\dots,x_{N}\right\}\subseteq\pazocal{X}$, the empirical Rademacher complexity of $\pazocal{F}$ with respect to $\pazocal{S}$ is defined as

[TABLE]

where $\sigma_{i}\sim\text{Uniform}\left(\left\{-1,1\right\}\right)$. Let $\pazocal{D}$ be a distribution over $\pazocal{X}$. Classic results from learning theory [31, 11] guarantee that with probability $1-\delta$ over the draw $\pazocal{S}\sim\pazocal{D}^{N}$, for every function $f\in\pazocal{F}$,

[TABLE]

Massart [35] proved the following Rademacher complexity bound for the case where the set $\left\{\left(f\left(x_{1}\right),\dots,f\left(x_{N}\right)\right):f\in\pazocal{F}\right\}$ is finite.

Lemma A.1**.**

[Massart [35]] Let $\pazocal{F}|_{\pazocal{S}}=\left\{\left(f\left(x_{1}\right),\dots,f\left(x_{N}\right)\right):f\in\pazocal{F}\right\}$. If $\left|\pazocal{F}|_{\pazocal{S}}\right|$ is finite, then $\widehat{\mathscr{R}}_{\pazocal{S}}(\pazocal{F})\leq\frac{r\sqrt{2\log\left|\pazocal{F}|_{\pazocal{S}}\right|}}{N}$, where $r=\max_{\bm{a}\in\pazocal{F}|_{\pazocal{S}}}||\bm{a}||_{2}$.

Appendix B Additional proofs and lemmas about Algorithm 1

The following Rademacher complexity bound guarantees that with probability $1-\zeta$, Algorithm 1 has a $\zeta$-representative run.

Lemma B.1**.**

For any $\tau\in\mathbb{Z}_{\geq 0}$, define the function classes $\pazocal{F}_{\tau}=\left\{j\mapsto\min\{\ell(\bm{\rho},j),\tau\}\mid\bm{\rho}\in\pazocal{P}\right\}$ and $\pazocal{H}_{\tau}=\left\{j\mapsto\textbf{1}_{\{\ell(\bm{\rho},j)\leq\tau\}}\mid\bm{\rho}\in\pazocal{P}\right\}$. Let $\pazocal{S}\subseteq\Pi$ be a set of problem instances. Then $\widehat{\mathscr{R}}_{\pazocal{S}}\left(\pazocal{F}_{\tau}\right)\leq\tau\sqrt{\frac{2d\ln\left|\textsc{Partition}(\pazocal{S},\tau)\right|}{|\pazocal{S}|}}$ and $\widehat{\mathscr{R}}_{\pazocal{S}}\left(\pazocal{H}_{\tau}\right)\leq\sqrt{\frac{2d\ln\left|\textsc{Partition}(\pazocal{S},\tau)\right|}{|\pazocal{S}|}}$.

Proof.

This lemma follows from Lemma A.1. ∎

Lemma B.2**.**

Suppose Algorithm 1 has a $\zeta$-representative run. For any set $\pazocal{P}^{\prime}\in\bar{\pazocal{G}}$ and all parameters $\bm{\rho}\in\pazocal{P}^{\prime}$, $t_{\delta/2}(\bm{\rho})\leq\tau_{\pazocal{P}^{\prime}}\leq t_{\delta/4}(\bm{\rho})$.

Proof.

We begin by proving that $t_{\delta/2}(\bm{\rho})\leq\tau_{\pazocal{P}^{\prime}}$.

Claim B.3**.**

Suppose Algorithm 1 has a $\zeta$-representative run. For any set $\pazocal{P}^{\prime}\in\pazocal{G}$ and all parameters $\bm{\rho}\in\pazocal{P}^{\prime}$, $t_{\delta/2}(\bm{\rho})\leq\tau_{\pazocal{P}^{\prime}}$.

Proof of Claim B.3.

Let $t$ be the round that $\pazocal{P}^{\prime}$ is added to $\pazocal{G}$. We know that for all parameter vectors $\bm{\rho}\in\pazocal{P}^{\prime}$,

[TABLE]

Since Algorithm 1 had a $\zeta$-representative run, we know that

[TABLE]

where the second inequality follows from the fact that $\tau_{\pazocal{P}^{\prime}}\leq 2^{t}$ and monotonicity. Based on Step 5 of Algorithm 1, we know that

[TABLE]

Moreover, by Equation (4), $\Pr_{j\sim\Gamma}\left[\ell(\bm{\rho},j)\leq\tau_{\pazocal{P}^{\prime}}\right]\geq 1-\frac{3\delta}{8}-\frac{1}{\left|\pazocal{S}_{t}\right|}-\eta\delta.$ Based on Step 5 of Algorithm 1, we also know that $\frac{1}{\sqrt{\left|\pazocal{S}_{t}\right|}}\leq\eta\delta$. Therefore, $\Pr_{j\sim\Gamma}\left[\ell(\bm{\rho},j)\leq\tau_{\pazocal{P}^{\prime}}\right]\geq 1-\left(\frac{3}{8}+\eta^{2}+\eta\right)\delta.$ Finally, since $\eta\leq\frac{1}{9}$, we have that $\Pr_{j\sim\Gamma}\left[\ell(\bm{\rho},j)\leq\tau_{\pazocal{P}^{\prime}}\right]>1-\delta/2$, which means that

[TABLE]

We claim that Equation (5) implies that $t_{\delta/2}(\bm{\rho})\leq\tau_{\pazocal{P}^{\prime}}$. For a contradiction, suppose $t_{\delta/2}(\bm{\rho})>\tau_{\pazocal{P}^{\prime}}$, or in other words, $t_{\delta/2}(\bm{\rho})\geq\tau_{\pazocal{P}^{\prime}}+1$. Since $t_{\delta/2}(\bm{\rho})=\textnormal{argmax}_{\tau\in\mathbb{Z}}\left\{\Pr_{j\sim\Gamma}[\ell(\bm{\rho},j)\geq\tau]\geq\delta/2\right\}$, this would mean that $\delta/2\leq\Pr_{j\sim\Gamma}\left[\ell(\bm{\rho},j)\geq t_{\delta/2}(\bm{\rho})\right]\leq\Pr_{j\sim\Gamma}\left[\ell(\bm{\rho},j)\geq\tau_{\pazocal{P}^{\prime}}+1\right]<\delta/2$, which is a contradiction. Therefore, the claim holds. ∎

Next, we prove that $\tau_{\pazocal{P}^{\prime}}\leq t_{\delta/4}(\bm{\rho})$.

Claim B.4**.**

Suppose Algorithm 1 has a $\zeta$-representative run. For any set $\pazocal{P}^{\prime}\in\pazocal{G}$ and all parameters $\bm{\rho}\in\pazocal{P}^{\prime}$, $\tau_{\pazocal{P}^{\prime}}\leq t_{\delta/4}(\bm{\rho})$.

Proof of Claim B.4.

Let $t$ be the round that $\pazocal{P}^{\prime}$ is added to $\pazocal{G}$. We know that for all parameter vectors $\bm{\rho}\in\pazocal{P}^{\prime}$,

[TABLE]

Therefore, $\frac{1}{\left|\pazocal{S}_{t}\right|}\sum_{j\in\pazocal{S}_{t}}\textbf{1}_{\left\{\ell\left(\bm{\rho},j\right)\leq\tau_{\pazocal{P}^{\prime}}-1\right\}}\leq 1-\frac{3\delta}{8}$. Since Algorithm 1 had a $\zeta$-representative run, we know that

[TABLE]

because $\tau_{\pazocal{P}^{\prime}}-1<\tau_{\pazocal{P}^{\prime}}\leq 2^{t}$ and monotonicity. Based on Step 5 of Algorithm 1,

[TABLE]

Since $\eta\leq 1/9$, we have that $\Pr_{j\sim\Gamma}\left[\ell(\bm{\rho},j)\leq\tau_{\pazocal{P}^{\prime}}-1\right]<1-\delta/4.$ Therefore, $\Pr_{j\sim\Gamma}\left[\ell(\bm{\rho},j)\geq\tau_{\pazocal{P}^{\prime}}\right]=\Pr_{j\sim\Gamma}\left[\ell(\bm{\rho},j)>\tau_{\pazocal{P}^{\prime}}-1\right]>\delta/4.$ Since

[TABLE]

we have that $\tau_{\pazocal{P}^{\prime}}\leq t_{\delta/4}(\bm{\rho})$. ∎

The lemma statement follows from Claims B.3 and B.4. ∎

Corollary B.5**.**

Suppose Algorithm 1 has a $\zeta$-representative run. For every set $\pazocal{P}^{\prime}\in\bar{\pazocal{G}}$ and any parameter vector $\bm{\rho}\in\pazocal{P}^{\prime}$, $\tau_{\pazocal{P}^{\prime}}\delta/4\leq\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell(\bm{\rho},j),\tau_{\pazocal{P}^{\prime}}\right\}\right].$

Proof.

By Lemma B.2, we know that $\tau_{\pazocal{P}^{\prime}}\leq t_{\delta/4}(\bm{\rho})$, so $\Pr_{j\sim\Gamma}\left[\ell(\bm{\rho},j)\geq\tau_{\pazocal{P}^{\prime}}\right]\geq\frac{\delta}{4}$. Therefore, $\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell(\bm{\rho},j),\tau_{\pazocal{P}^{\prime}}\right\}\right]\geq\tau_{\pazocal{P}^{\prime}}\Pr_{j\sim\Gamma}\left[\ell(\bm{\rho},j)\geq\tau_{\pazocal{P}^{\prime}}\right]\geq\frac{\tau_{\pazocal{P}^{\prime}}\delta}{4}.$ ∎

See 4.5

Proof.

The last round that Algorithm 1 adds any subset to the set $\pazocal{G}$ is round $\bar{t}-1$. For ease of notation, let $\bar{\pazocal{S}}=\pazocal{S}_{\bar{t}-1}$. Since $\bm{\rho}$ is not an element of any set in $\pazocal{G}$, the cap $2^{\bar{t}-1}$ must be too small compared to the average loss of the parameter $\bm{\rho}$. Specifically, it must be that $\frac{1}{\left|\bar{\pazocal{S}}\right|}\sum_{j\in\bar{\pazocal{S}}}\textbf{1}_{\left\{\ell\left(\bm{\rho},j\right)\leq 2^{\bar{t}-1}\right\}}<1-\frac{3\delta}{8}$. Otherwise, the algorithm would have added a parameter set containing $\bm{\rho}$ to the set $\pazocal{G}$ on round $\bar{t}-1$ (Step 7). Since Algorithm 1 had a $\zeta$-representative run, we know that the probability the loss of $\bm{\rho}$ is smaller than $2^{\bar{t}-1}$ converges to the fraction of samples with loss smaller than $2^{\bar{t}-1}$. Specifically,

[TABLE]

where the second inequality follows from Step 5 of Algorithm 1. Using our bound of $1-3\delta/8$ on the fraction of samples with loss smaller than $2^{\bar{t}-1}$, we have that $\Pr_{j\sim\Gamma}\left[\ell(\bm{\rho},j)\leq 2^{\bar{t}-1}\right]<1-(3/8-\eta)\delta$. Since $\eta\leq 1/9$, it must be that $\Pr_{j\sim\Gamma}\left[\ell(\bm{\rho},j)\leq 2^{\bar{t}-1}\right]<1-\delta/4$, or conversely, $\Pr_{j\sim\Gamma}\left[\ell(\bm{\rho},j)\geq 2^{\bar{t}-1}\right]\geq\Pr_{j\sim\Gamma}\left[\ell(\bm{\rho},j)>2^{\bar{t}-1}\right]>\delta/4$. Since

[TABLE]

we have that $2^{\bar{t}-1}\leq t_{\delta/4}(\bm{\rho})$. Therefore,

[TABLE]

∎

Lemma B.6**.**

Suppose Algorithm 1 has a $\zeta$-representative run. For any parameter vector $\bm{\rho}\not\in\bigcup_{\pazocal{P}^{\prime}\in\bar{\pazocal{G}}}\pazocal{P}^{\prime}$, $\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell(\bm{\rho},j),t_{\delta/4}(\bm{\rho})\right\}\right]\geq\bar{T}.$

Proof.

From Lemma 4.5, we know that $\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell(\bm{\rho},j),t_{\delta/4}(\bm{\rho})\right\}\right]\geq 2^{\bar{t}-3}\delta.$ Moreover, from Step 3 of Algorithm 1, $\bar{T}\leq 2^{\bar{t}-3}\delta$, so $\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell(\bm{\rho},j),t_{\delta/4}(\bm{\rho})\right\}\right]\geq\bar{T}.$ ∎

Lemma B.7**.**

Suppose Algorithm 1 has a $\zeta$-representative run. There exists a set $\pazocal{P}^{\prime}\in\bar{\pazocal{G}}$ and a parameter vector $\bm{\rho}\in\pazocal{P}^{\prime}$ such that $\bar{T}\geq\frac{1}{\sqrt[4]{1+\epsilon}}\cdot\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell(\bm{\rho},j),\tau_{\pazocal{P}^{\prime}}\right\}\right]$.

Proof.

By definition of the upper confidence bound $\bar{T}$ (Steps 9 through 11 of Algorithm 1), there is some round $t$, some set $\pazocal{P}^{\prime}\in\bar{\pazocal{G}}$, and some parameter vector $\bm{\rho}\in\pazocal{P}^{\prime}$ such that $\bar{T}=\frac{1}{\left|\pazocal{S}_{t}\right|}\sum_{j\in\pazocal{S}_{t}}\min\left\{\ell\left(\bm{\rho},j\right),\tau_{\pazocal{P}^{\prime}}\right\}$. Since Algorithm 1 had a $\zeta$-representative run,

[TABLE]

By Step 7, we know that at least a $(1-3\delta/8)$-fraction of the problem instances $j\in\pazocal{S}_{t}$ have a loss $\ell(\bm{\rho},j)$ that is at most $2^{t}$. Therefore, by definition of $\tau_{\pazocal{P}^{\prime}}=\tau_{\lfloor|\pazocal{S}_{t}|\left(1-3\delta/8\right)\rfloor}$, it must be that $\tau_{\pazocal{P}^{\prime}}\leq 2^{t}$. By monotonicity, this means that

[TABLE]

Based on Step 5 of Algorithm 1,

[TABLE]

From Corollary B.5, $\tau_{\pazocal{P}^{\prime}}\delta/4\leq\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell(\bm{\rho},j),\tau_{\pazocal{P}^{\prime}}\right\}\right]$, which means that

[TABLE]

Finally, the lemma statement follows from the fact that

[TABLE]

∎

Lemma B.8**.**

Suppose Algorithm 1 has a $\zeta$-representative run. For every set $\pazocal{P}^{\prime}\in\bar{\pazocal{G}}$ and every pair of parameter vectors $\bm{\rho}_{1},\bm{\rho}_{2}\in\pazocal{P}^{\prime}$, $\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell\left(\bm{\rho}_{1},j\right),\tau_{\pazocal{P}^{\prime}}\right\}\right]\leq\sqrt[4]{1+\epsilon}\cdot\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell\left(\bm{\rho}_{2},j\right),\tau_{\pazocal{P}^{\prime}}\right\}\right]$.

Proof.

Let $t$ be the round that the interval $\pazocal{P}^{\prime}$ was added to $\pazocal{G}$. Since Algorithm 1 had a $\zeta$-representative run,

[TABLE]

By definition of the set $\pazocal{P}^{\prime}$, for all problem instances $j\in\pazocal{S}_{t}$, $\min\left\{\ell\left(\bm{\rho}_{1},j\right),\tau_{\pazocal{P}^{\prime}}\right\}=\min\left\{\ell\left(\bm{\rho}_{2},j\right),\tau_{\pazocal{P}^{\prime}}\right\}.$ Therefore,

[TABLE]

Again, since Algorithm 1 had a $\zeta$-representative run,

[TABLE]

Since $\tau_{\pazocal{P}^{\prime}}\leq 2^{t}$,

[TABLE]

Based on Step 5 of Algorithm 1, this means that

[TABLE]

By Corollary B.5, $\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell\left(\bm{\rho}_{1},j\right),\tau_{\pazocal{P}^{\prime}}\right\}\right]\leq(1+8\eta)\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell\left(\bm{\rho}_{2},j\right),\tau_{\pazocal{P}^{\prime}}\right\}\right]$. The lemma statement follows from the fact that $\eta\leq\left(\sqrt[4]{1+\epsilon}-1\right)/8$. ∎

See 4.6

Proof.

Since $OPT_{\delta/4}:=\inf_{\bm{\rho}\in\pazocal{P}}\left\{\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell\left(\bm{\rho},j\right),t_{\delta/4}(\bm{\rho})\right\}\right]\right\}$, we know that for any $\gamma>0$, there exists a parameter vector $\bm{\rho}\in\pazocal{P}$ such that $\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell(\bm{\rho},j),t_{\delta/4}(\bm{\rho})\right\}\right]\leq OPT_{\delta/4}+\gamma.$ We claim there exists a parameter $\bm{\rho}^{\prime}\in\pazocal{P}^{*}$ such that

[TABLE]

and thus the lemma statement holds (see Lemma B.9).

First, suppose $\bm{\rho}$ is contained in a set $\pazocal{P}^{\prime}\in\bar{\pazocal{G}}$. By Lemmas B.2 and B.8, there exists a parameter $\bm{\rho}^{\prime}\in\pazocal{P}^{\prime}\cap\pazocal{P}^{*}$ such that

[TABLE]

Since $\sqrt[4]{1+\epsilon}\leq\sqrt{1+\epsilon}$, Equation (6) holds in this case.

Otherwise, suppose $\bm{\rho}\not\in\bigcup_{\pazocal{P}^{\prime}\in\bar{\pazocal{G}}}\pazocal{P}^{\prime}$. By Lemma B.6, we know that $\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell(\bm{\rho},j),t_{\delta/4}(\bm{\rho})\right\}\right]\geq\bar{T}.$ Moreover, by Lemma B.7, there exists a set $\pazocal{P}^{\prime}\in\bar{\pazocal{G}}$ and parameter vector $\bm{\rho}^{*}\in\pazocal{P}^{\prime}$ such that $\sqrt[4]{1+\epsilon}\cdot\bar{T}\geq\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell\left(\bm{\rho}^{*},j\right),\tau_{\pazocal{P}^{\prime}}\right\}\right]$. Finally, by Lemma B.8, there exists a parameter vector $\bm{\rho}^{\prime}\in\pazocal{P}^{\prime}\cap\pazocal{P}^{*}$ such that

[TABLE]

Therefore, Equation (6) holds in this case as well. ∎

Lemma B.9**.**

Let $\pazocal{P}^{*}$ be the set of parameters output by Algorithm 1. Suppose that for every $\gamma>0$, there exists a parameter vector $\bm{\rho}^{\prime}\in\pazocal{P}^{*}$ such that $\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell(\bm{\rho}^{\prime},j),t_{\delta/2}(\bm{\rho}^{\prime})\right\}\right]\leq\sqrt{1+\epsilon}\left(OPT_{\delta/4}+\gamma\right).$ Then $\min_{\bm{\rho}\in\pazocal{P}^{*}}\left\{\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell(\bm{\rho},j),t_{\delta/2}(\bm{\rho})\right\}\right]\right\}\leq\sqrt{1+\epsilon}\cdot OPT_{\delta/4}.$

Proof.

For a contradiction, suppose that $\min_{\bm{\rho}\in\pazocal{P}^{*}}\left\{\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell(\bm{\rho},j),t_{\delta/2}(\bm{\rho})\right\}\right]\right\}>\sqrt{1+\epsilon}\cdot OPT_{\delta/4}$ and let $\gamma^{\prime}=\min_{\bm{\rho}\in\pazocal{P}^{*}}\left\{\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell(\bm{\rho},j),t_{\delta/2}(\bm{\rho})\right\}\right]\right\}-\sqrt{1+\epsilon}\cdot OPT_{\delta/4}.$ Setting $\gamma=\frac{\gamma^{\prime}}{2\sqrt{1+\epsilon}}$, we know there exists a parameter vector $\bm{\rho}^{\prime}\in\pazocal{P}^{*}$ such that

[TABLE]

which is a contradiction. ∎

See 4.4

Proof.

For each set $\pazocal{P}^{\prime}\in\bar{\pazocal{G}}$, let $t_{\pazocal{P}^{\prime}}$ be the round where $\pazocal{P}^{\prime}$ is added to the set $\pazocal{G}$, let $\pazocal{S}_{\pazocal{P}^{\prime}}=\pazocal{S}_{t_{\pazocal{P}^{\prime}}}$, and let $\bm{\rho}_{\pazocal{P}^{\prime}}$ be an arbitrary parameter vector in $\pazocal{P}^{\prime}$. Since no set is added to $\pazocal{G}$ when $t=\bar{t}$, it must be that for all sets $\pazocal{P}^{\prime}\in\bar{\pazocal{G}}$, $t_{\pazocal{P}^{\prime}}\leq\bar{t}-1$. Moreover, since $OPT_{\delta/4}:=\inf_{\bm{\rho}\in\pazocal{P}}\left\{\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell\left(\bm{\rho},j\right),t_{\delta/4}(\bm{\rho})\right\}\right]\right\}$, we know that for every $\gamma>0$, there exists a parameter vector $\bm{\rho}^{*}$ such that

[TABLE]

Below, we prove that $2^{\bar{t}}\leq\frac{16\sqrt[4]{1+\epsilon}}{\delta}\cdot\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell\left(\bm{\rho}^{*},j\right),t_{\delta/4}\left(\bm{\rho}^{*}\right)\right\}\right]$ and thus the lemma statement holds (see Lemma B.10).

Case 1: $\bm{\rho}^{*}\not\in\bigcup_{\pazocal{P}^{\prime}\in\bar{\pazocal{G}}}\pazocal{P}^{\prime}$.

By Lemma 4.5, we know that $2^{\bar{t}-3}\delta\leq\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell\left(\bm{\rho}^{*},j\right),t_{\delta/4}\left(\bm{\rho}^{*}\right)\right\}\right]$. Therefore, $2^{\bar{t}}\leq\frac{8}{\delta}\cdot\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell\left(\bm{\rho}^{*},j\right),t_{\delta/4}\left(\bm{\rho}^{*}\right)\right\}\right]\leq\frac{16\sqrt[4]{1+\epsilon}}{\delta}\cdot\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell\left(\bm{\rho}^{*},j\right),t_{\delta/4}\left(\bm{\rho}^{*}\right)\right\}\right]$.

Case 2: $\bm{\rho}^{*}$ is an element of a set $\pazocal{P}^{\prime}\in\bar{\pazocal{G}}$ and $t_{\pazocal{P}^{\prime}}\leq\bar{t}-2$.

Let $T^{\prime}$ be the value of $T$ at the beginning of round $\bar{t}-1$. Since the algorithm does not terminate on round $\bar{t}-1$, it must be that $2^{\bar{t}-4}\delta<T^{\prime}$. By definition of $T^{\prime}$,

[TABLE]

Since Algorithm 1 had a $\zeta$-representative run, $2^{\bar{t}-4}\delta$ is upper-bounded by

[TABLE]

Since $\tau_{\pazocal{P}^{\prime}}\leq 2^{t_{\pazocal{P}^{\prime}}}$ and $f$ is monotone, $2^{\bar{t}-4}\delta$ is at most

[TABLE]

By Step 5 of Algorithm 1, $2^{\bar{t}-4}\delta\leq\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell\left(\bm{\rho}^{*},j\right),\tau_{\pazocal{P}^{\prime}}\right\}\right]+\tau_{\pazocal{P}^{\prime}}\eta\delta.$ Finally, by Corollary B.5, $2^{\bar{t}-4}\delta\leq(1+4\eta)\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell\left(\bm{\rho}^{*},j\right),\tau_{\pazocal{P}^{\prime}}\right\}\right].$ Since $\eta<\left(\sqrt[4]{1+\epsilon}-1\right)/4$,

[TABLE]

Recalling that $\tau_{\pazocal{P}^{\prime}}\leq t_{\delta/4}\left(\bm{\rho}^{*}\right)$ by Lemma B.2, we conclude that

[TABLE]

Case 3: $\bm{\rho}^{*}$ is not an element of any set $\bar{\pazocal{P}}\in\bar{\pazocal{G}}$ with $t_{\bar{\pazocal{P}}}\leq\bar{t}-2$, but

$\bm{\rho}^{*}$ is an element of a set $\pazocal{P}^{\prime}\in\bar{\pazocal{G}}$ with $t_{\pazocal{P}^{\prime}}=\bar{t}-1$.

Let $\pazocal{S}^{\prime}=\pazocal{S}_{\bar{t}-2}$ and let $\bar{\pazocal{P}}$ be the set containing $\bm{\rho}^{*}$ in Step 6 on round $\bar{t}-2$. Since $\bar{\pazocal{P}}$ was not added to $\pazocal{G}$ on round $\bar{t}-2$, we know that fewer than a $\left(1-\frac{3\delta}{8}\right)$-fraction of the instances in $\pazocal{S}^{\prime}$ have a loss of at most $2^{\bar{t}-2}$ when run with any parameter vector $\bm{\rho}\in\bar{\pazocal{P}}$ (including $\bm{\rho}^{*}$). In other words,

[TABLE]

Since Algorithm 1 had a $\zeta$-representative run,

[TABLE]

Based on Step 5 of Algorithm 1, $\Pr_{j\sim\Gamma}\left[\ell\left(\bm{\rho}^{*},j\right)\leq 2^{\bar{t}-2}\right]<1-(3/8-\eta)\delta$. Since $\eta\leq 1/9$, $\Pr_{j\sim\Gamma}\left[\ell\left(\bm{\rho}^{*},j\right)\leq 2^{\bar{t}-2}\right]<1-\delta/4$. Therefore, $\Pr_{j\sim\Gamma}\left[\ell\left(\bm{\rho}^{*},j\right)\geq 2^{\bar{t}-2}\right]\geq\Pr_{j\sim\Gamma}\left[\ell\left(\bm{\rho}^{*},j\right)>2^{\bar{t}-2}\right]\geq\delta/4$. Since $t_{\delta/4}(\bm{\rho}^{*})=\textnormal{argmax}_{\tau\in\mathbb{Z}}\left\{\Pr_{j\sim\Gamma}[\ell(\bm{\rho}^{*},j)\geq\tau]\geq\delta/4\right\}$, we have that $2^{\bar{t}-2}\leq t_{\delta/4}\left(\bm{\rho}^{*}\right)$. Therefore,

[TABLE]

which means that $2^{\bar{t}}\leq\frac{16}{\delta}\cdot\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell\left(\bm{\rho}^{*},j\right),t_{\delta/4}\left(\bm{\rho}^{*}\right)\right\}\right]$. ∎

Lemma B.10**.**

Suppose that for all $\gamma>0$, there exists a parameter vector $\bm{\rho}^{*}\in\pazocal{P}$ such that $2^{\bar{t}}\leq\frac{16\sqrt[4]{1+\epsilon}}{\delta}\cdot\mathop{\mathbb{E}}_{j\sim\Gamma}\left[\min\left\{\ell\left(\bm{\rho}^{*},j\right),t_{\delta/4}\left(\bm{\rho}^{*}\right)\right\}\right]\leq\frac{16\sqrt[4]{1+\epsilon}}{\delta}\left(OPT_{\delta/4}+\gamma\right)$. Then $2^{\bar{t}}\leq\frac{16\sqrt[4]{1+\epsilon}}{\delta}\cdot OPT_{\delta/4}$.

Proof.

For a contradiction, suppose $2^{\bar{t}}>\frac{16\sqrt[4]{1+\epsilon}}{\delta}\cdot OPT_{\delta/4}$ and let $\gamma^{\prime}=2^{\bar{t}}-\frac{16\sqrt[4]{1+\epsilon}}{\delta}\cdot OPT_{\delta/4}$. Letting $\gamma=\frac{\gamma^{\prime}\delta}{32\sqrt[4]{1+\epsilon}}$, we know there exists a parameter vector $\bm{\rho}^{*}\in\pazocal{P}$ such that

[TABLE]

which is a contradiction. Therefore, the lemma statement holds. ∎

Lemma B.11**.**

Suppose Algorithm 1 has a $\zeta$-representative run. The size of the set $\pazocal{P}^{*}\subset\pazocal{P}$ that Algorithm 1 returns is bounded by $\sum_{t=1}^{\bar{t}}\left|\textsc{Partition}\left(\pazocal{S}_{t},\frac{16}{\delta}\sqrt[4]{1+\epsilon}\cdot OPT_{\delta/4}\right)\right|$.

Proof.

Based on Step 13 of Algorithm 1, the size of $\pazocal{P}^{*}$ equals the size of the set $\bar{\pazocal{G}}$. Algorithm 1 only adds sets to $\pazocal{G}$ on Step 8, and on each round $t$, the number of sets it adds is bounded by $\left|\textsc{Partition}\left(\pazocal{S}_{t},2^{t}\right)\right|$. By monotonicity, we know that $\left|\textsc{Partition}\left(\pazocal{S}_{t},2^{t}\right)\right|\leq\left|\textsc{Partition}\left(\pazocal{S}_{t},2^{\bar{t}}\right)\right|\leq\left|\textsc{Partition}\left(\pazocal{S}_{t},\frac{16}{\delta}\sqrt[4]{1+\epsilon}\cdot OPT_{\delta/4}\right)\right|$. Therefore,

[TABLE]

∎

Appendix C Additional information about related research

In this section, we begin by surveying additional related research. We then describe the bounds on $\left|\textsc{Partition}(\pazocal{S},\tau)\right|$ that prior research has provided in the contexts of integer programming and clustering. We also describe an integer programming algorithm configuration problem where the best-known uniform convergence bound is exponential in the number of variables [6], whereas our algorithm only requires $\tilde{O}\left(\left(\delta\eta\right)^{-2}\right)$ samples.

C.1 Survey of additional related research

A related line of research [22, 18, 7, 10, 2] studies algorithm configuration in online learning settings, where the learner encounters a sequence—perhaps adversarially selected—of problem instances over a series of timesteps. The learner’s goal is to select an algorithm on each timestep so that the learner has strong cumulative performance across all timesteps (as quantified by regret, typically). In contrast, this paper is focused on the batch learning setting, where the problem instances are not adversarially generated but come from a fixed distribution.

A number of papers have explored Bayesian optimization as a tool for parameter optimization. These algorithms have strong performance in practice [12, 25, 26, 38], but often do not come with theoretical guarantees, which is the focus of our paper. Those papers that do include provable guarantees typically require that the loss function is smooth, as quantified by its Lipschitz constant [16] or its RKHS norm [13], which is not the case in our setting.

C.2 Integer programming

In this section, to match prior research, we use the notation $Q$ to denote an integer program (rather than the notation $j$, as in the main body).

Balcan et al. [6] study mixed integer linear programs (MILPs) where the goal is to maximize an objective function $\bm{c}^{\top}\bm{x}$ subject to the constraints that $A\bm{x}\leq\bm{b}$ and that some of the components of $\bm{x}$ are contained in $\{0,1\}$. Given a MILP $Q$, we use the notation $\breve{\bm{x}}_{Q}=\left(\breve{x}_{Q}[1],\dots\breve{x}_{Q}[n]\right)$ to denote an optimal solution to the MILP’s LP relaxation. We denote the optimal objective value to the MILP’s LP relaxation as $\breve{c}_{Q}$, which means that $\breve{c}_{Q}=\bm{c}^{\top}\breve{\bm{x}}_{Q}$.

The most popular algorithm for solving MILPs is called branch-and-bound (B&B), which we now describe at a high level. Let $Q^{\prime}$ be a MILP we want to solve. B&B builds a search tree $\mathscr{T}$ with $Q^{\prime}$ at the root. At each round, the algorithm uses a node selection policy (such as depth- or best-first search) to choose a leaf of $\mathscr{T}$. This leaf node corresponds to a MILP we denote as $Q$. Using a variable selection policy, the algorithm then chooses one of that MILP’s variables. Specifically, let $Q^{+}_{i}$ (resp., $Q^{-}_{i}$) equal the MILP $Q$ after adding the constraint $x_{i}=1$ (resp., $x_{i}=0$). The algorithm defines the right (resp., left) child of the leaf $Q$ to equal $Q^{+}_{i}$ (resp., $Q^{-}_{i}$). B&B then tries to “fathom” these leafs. At a high level, B&B fathoms a leaf if it can guarantee that it will not find any better solution by branching on that leaf than the best solution found so far. See, for example, the research by Balcan et al. [6] for the formal protocol. Once B&B has fathomed every leaf, it terminates. It returns the best feasible solution to $Q^{\prime}$ that it found in the search tree, which is provably optimal.

Balcan et al. [6] focus on variable selection policies, and in particular, score-based variable selection policies, defined below.

Definition C.1** (Score-based variable selection policy [6]).**

Let score be a deterministic function that takes as input a partial search tree $\mathscr{T}$, a leaf $Q$ of that tree, and an index $i$, and returns a real value $\textnormal{{score}}(\mathscr{T},Q,i)\in\mathbb{R}$. For a leaf $Q$ of a tree $\mathscr{T}$, let $N_{\mathscr{T},Q}$ be the set of variables that have not yet been branched on along the path from the root of $\mathscr{T}$ to $Q$. A score-based variable selection policy selects the variable $\textnormal{argmax}_{x_{i}\in N_{\mathscr{T},Q}}\{\textnormal{{score}}(\mathscr{T},Q,i)\}$ to branch on at the node $Q$.

Score-based variable selection policies are extremely popular in B&B implementations [33, 1, 21]. See the research by Balcan et al. [6] for examples. Given $d$ arbitrary scoring rules $\textnormal{{score}}_{1},\dots,\textnormal{{score}}_{d}$, Balcan et al. [6] provide guidance for learning a linear combination $\rho_{1}\textnormal{{score}}_{1}+\cdots+\rho_{d}\textnormal{{score}}_{d}$ that leads to small expected tree sizes. They assume that all aspects of the tree search algorithm except the variable selection policy, such as the node selection policy, are fixed. In their analysis, they prove the following lemma.

Lemma C.2**.**

[Balcan et al. [6]] Let $\textnormal{{score}}_{1},\dots,\textnormal{{score}}_{d}$ be $d$ arbitrary scoring rules and let $Q$ be an arbitrary MILP over $n$ binary variables. Suppose we limit B&B to producing search trees of size $\tau$. There is a set $\mathscr{H}$ of at most $n^{2(\tau+1)}$ hyperplanes such that for any connected component $R$ of $[0,1]^{d}\setminus\mathscr{H}$, the search tree B&B builds using the scoring rule $\rho_{1}\textnormal{{score}}_{1}+\cdots+\rho_{d}\textnormal{{score}}_{d}$ is invariant across all $(\rho_{1},\dots,\rho_{d})\in R$.

Balcan et al. [6] observe that in practice, the number of hyperplanes is significantly smaller. Given a set $\pazocal{S}$ of MILP instances, there are $|\pazocal{S}|n^{2(\tau+1)}$ relevant hyperplanes $\pazocal{H}^{*}$. The number of connected components of the set $[0,1]^{d}\setminus\pazocal{H}^{*}$ is at most $\left(|\pazocal{S}|n^{2(\tau+1)}+1\right)^{d}$ [17]. Therefore, in our context, $\left|\textsc{Partition}(\pazocal{S},\tau)\right|\leq\left(|\pazocal{S}|n^{2(\tau+1)}+1\right)^{d}$. When $d=2$, Balcan et al. [6] provide guidance for finding the partition of $[0,1]$ into intervals $I$ where the search tree B&B builds using the scoring rule $\rho\cdot\textnormal{{score}}_{1}+(1-\rho)\cdot\textnormal{{score}}_{2}$ is invariant across all $\rho\in I$. These intervals correspond to the output of the function Partition. An important direction for future research is extending the implementation to multi-dimensional parameter spaces.

Uniformly sampling configurations.

See 5.2

Proof.

We know that the probability $\left\{\rho_{1},\dots,\rho_{m}\right\}\cap(a,b)=\emptyset$ is $(1-(b-a))^{m}\geq(1-(b-a))^{1/(b-a)}\geq\frac{1}{3}$ since $b-a\leq\frac{1}{6}$. ∎

Uniform convergence versus Algorithm 1.

We now describe an integer programming algorithm configuration problem where the best-known uniform convergence bound is exponential in the number of variables [6], whereas our algorithm only requires $\tilde{O}\left(\left(\delta\eta\right)^{-2}\right)$ samples. We use a family of MILP distributions introduced by Balcan et al. [6]:

Theorem C.3** (Balcan et al. [6]).**

For any MILP $Q$, let

[TABLE]

and

[TABLE]

Define $\ell(\rho,Q)$ to be the size of the tree B&B produces using the scoring rule $\rho\cdot\textnormal{{score}}_{1}+(1-\rho)\cdot\textnormal{{score}}_{2}$ given $Q$ as input. For every $a,b$ such that $\frac{1}{3}<a<b<\frac{1}{2}$ and for all even $n\geq 6$, there exists an infinite family of distributions $\pazocal{D}$ over MILP instances with $n$ variables such that the following conditions hold:

If $\rho\leq a$, then $\ell(\rho,Q)=2^{(n-5)/4}$ with probability $\frac{1}{2}$ and $\ell(\rho,Q)=8$ with probability $\frac{1}{2}$. 2. 2.

If $\rho\in(a,b)$, then $\ell(\rho,Q)=8$ with probability $1$. 3. 3.

If $\rho\geq b$, then $\ell(\rho,Q)=2^{(n-4)/2}$ with probability $\frac{1}{2}$ and $\ell(\rho,Q)=8$ with probability $\frac{1}{2}$.

This holds no matter which node selection policy B&B uses.

As we describe in Section 5, the sample complexity bound Balcan et al. [6] provide implies that at least $\left(\frac{2^{n}}{\epsilon^{\prime}}\right)^{2}$ samples are required to ensure that the average tree size branch-and-bound builds over the samples is within an additive $\epsilon^{\prime}$ factor of the expected tree size. Even for $\epsilon^{\prime}=2^{n/2}$, this sample complexity bound is exponential in $n$.

See 5.3

Proof.

In the proof of Theorem C.3, Balcan et al. [6] show that for any distribution $\pazocal{D}$ in this family and any subset $\pazocal{S}$ from the support of $\pazocal{D}$, $\left|\textsc{Partition}(\pazocal{S},\tau)\right|=3$, and the partition Partition returns is $[0,a]$, $(a,b)$, and $[b,1]$. Therefore, for the first three iterations, our algorithm will use $\tilde{O}\left(\left(\delta\eta\right)^{-2}\right)$ samples. At that point, $t=3$, and the tree-size cap is 8. Algorithm 1 will discover that for all of the samples $Q\in\pazocal{S}_{3}$, when $\rho\in(a,b)$, $\ell(\rho,Q)=8$. Therefore, it add $(a,b)$ to $\pazocal{G}$ and it will set $T=8$. It will continue drawing $\tilde{O}\left(\left(\delta\eta\right)^{-2}\right)$ samples at each round until $2^{t-3}\delta\geq T=8$, or in other words, until $t=O(\log(1/\delta))$. Therefore, the total number of samples it draws is $\tilde{O}\left(\left(\delta\eta\right)^{-2}\right)$. The set it returns will contain a point $\rho\in(a,b)$, which is optimal. ∎

C.3 Clustering

We begin with an overview of agglomerative clustering algorithms. A clustering instance $(V,d)$ consists of a set $V$ of $n$ points and a distance metric $d:V\times V\to\mathbb{R}_{\geq 0}$ specifying all pairwise distances between these points. The goal is to partition the points into groups such that distances within each group are minimized and distances between each group are maximized. Typically, the quality of a clustering is measured by an objective function, such as the classic $k$-means, $k$-median, or $k$-center objectives. Unfortunately, it is NP-hard to determine the clustering that minimizes any of these objectives.

An agglomerative clustering algorithm is characterized by a merge function ${\xi}(A,B)\to\mathbb{R}_{\geq 0}$, which defines the distance between any two sets of points $A,B\subseteq V$. The algorithm builds a cluster tree $\mathscr{T}$, starting with $n$ singleton leaf nodes, each of which contains one point from $V$. The algorithm iteratively merges the two sets with minimum distance until there is a single node remaining, consisting of the set $V$. The children of any node $N$ in this tree correspond to the two sets of points that were merged to form $N$. Common choices for the merge function ${\xi}$ include $\min_{a\in A,b\in B}d(a,b)$ (single-linkage), $\frac{1}{|A|\cdot|B|}\sum_{a\in A,b\in B}d(a,b)$ (average-linkage) and $\max_{a\in A,b\in B}d(a,b)$ (complete-linkage). The linkage procedure is followed by a dynamic programming step, which returns the pruning of the tree that minimizes a fixed objective function, such as the $k$-means, $k$-median, or $k$-center objectives. If the linkage procedure is terminated early, we will be left with a forest, rather than a single tree. The dynamic programming procedure can easily be adjusted to find the best pruning of the forest (for example, by completing the hierarchy arbitrarily and enforcing that the dynamic programming algorithm only return clusters contained in the original forest).

Balcan et al. [5] define three infinite families of merge functions. The families $\pazocal{A}_{1}$ and $\pazocal{A}_{2}$ consist of merge functions ${\xi}(A,B)$ that depend on the minimum and maximum of all pairwise distances between $A$ and $B$. The second family, denoted by $\mathscr{A}_{3}$, depends on all pairwise distances between $A$ and $B$. All classes are parameterized by a single value $\rho$.

[TABLE]

The classes $\pazocal{A}_{1}$ and $\pazocal{A}_{2}$ define spectra of merge functions ranging from single-linkage ($\xi_{1,-\infty}$ and $\xi_{2,1}$) to complete-linkage ($\xi_{1,\infty}$ and $\xi_{2,0}$). The class $\pazocal{A}_{3}$ includes average-, complete-, and single-linkage.

Lemma C.4**.**

Let $(V,d)$ be an arbitrary clustering instance over $n$ points. There is a partition of $\mathbb{R}$ into $k=O(n^{8})$ intervals $I_{1},\dots,I_{k}$ such that for any interval $I_{i}$ and any two parameters $\rho,\rho^{\prime}\in I_{i}$, the sequences of merges the agglomerative clustering algorithm makes using the merge functions $\xi_{1,\rho}$ and $\xi_{1,\rho^{\prime}}$ are identical. The same holds for the set of merge functions $\pazocal{A}_{2}$.

Given a set $\pazocal{S}$ of clustering instances, there are $O\left(|\pazocal{S}|n^{8}\right)$ relevant intervals. Therefore, in our context, $\left|\textsc{Partition}(\pazocal{S},\tau)\right|=O(|\pazocal{S}|n^{8})$. Balcan et al. [5] provide guidance for finding these intervals, which correspond to the output of the function Partition. They also prove the following guarantee for the class $\pazocal{A}_{3}$.

Lemma C.5**.**

Let $(V,d)$ be an arbitrary clustering instance over $n$ points. There is a partition of $\mathbb{R}$ into $k=O\left(n^{2}3^{2n}\right)$ intervals $I_{1},\dots,I_{k}$ such that for any interval $I_{i}$ and any two parameters $\rho,\rho^{\prime}\in I_{i}$, the sequences of merges the agglomerative clustering algorithm makes using the merge functions $\xi_{3,\rho}$ and $\xi_{3,\rho^{\prime}}$ are identical.

Similarly, given a set $\pazocal{S}$ of clustering instances, there are $O\left(|\pazocal{S}|n^{2}3^{2n}\right)$ relevant intervals. Therefore, in our context, $\left|\textsc{Partition}(\pazocal{S},\tau)\right|=O\left(|\pazocal{S}|n^{2}3^{2n}\right)$. Again, Balcan et al. [5] provide guidance for finding these intervals.