Adaptive Exact Learning of Decision Trees from Membership Queries

TL;DR

This paper advances the adaptive learning of decision trees from membership queries by providing more query-efficient algorithms, improving upon previous randomized and deterministic methods for trees of bounded depth.

Contribution

It introduces new randomized and deterministic algorithms with reduced query complexity for learning decision trees of bounded depth from membership queries.

Findings

Improved randomized algorithm with $ ilde O(2^{2d}) + 2^{d} ext{log} n$ queries.

Enhanced deterministic algorithm with $2^{5.83d}+2^{2d+o(d)} ext{log} n$ queries.

Both algorithms operate in polynomial time and outperform prior methods.

Abstract

In this paper we study the adaptive learnability of decision trees of depth at most $d$ from membership queries. This has many applications in automated scientific discovery such as drugs development and software update problem. Feldman solves the problem in a randomized polynomial time algorithm that asks $\tilde O(2^{2d})\log n$ queries and Kushilevitz-Mansour in a deterministic polynomial time algorithm that asks $ 2^{18d+o(d)}\log n$ queries. We improve the query complexity of both algorithms. We give a randomized polynomial time algorithm that asks $\tilde O(2^{2d}) + 2^{d}\log n$ queries and a deterministic polynomial time algorithm that asks $2^{5.83d}+2^{2d+o(d)}\log n$ queries.

Tables

Table 1. Table 1: Query complexity for polynomial time adaptive algorithms

Deterministic/	Algorithm	Query
Randomized		Complexity
Randomized (1)	[19, 13]	$\tilde{O}\left(2^{2d}\right)\log n$
Randomized	Ours	$\tilde{O}\left(2^{2d}\right)+2^d\log n$
Deterministic(2)	[20, 21, 23]	$2^{18d+o\left(d\right)}\log n$
Deterministic	Ours	$2^{5.83d}+2^{2d+o\left(d\right)}\log n$

Equations

$$\tilde{O}(2^{2d})\log n\ge {\mathrm{OPT}}_{\mathrm{NAD}}({\mathrm{DT}}_d)\ge {\mathrm{OPT}}_{\mathrm{AD}}({\mathrm{DT}}_d)\ge \Omega (2^d\log n)$$

$$\tilde{O}(2^{2d})\log n\ge {\mathrm{OPT}}_{\mathrm{R},\mathrm{NAD}}({\mathrm{DT}}_d)\ge {\mathrm{OPT}}_{\mathrm{R},\mathrm{AD}}({\mathrm{DT}}_d)\ge \Omega (2^d\log n).$$

$$O\left(\frac{d^2\log n}{\log (q/(2d^2))}\right),$$

$$g^{(i)}(a)=g(a_1,\cdots ,a_{i-1},a_1,a_{i+1},\cdots ,a_n)\ne g(a).$$

$$Pr\left[\left|\frac{1}{m}{\sum }_{i=1}^m{X}_i-\mu \right|\ge \lambda \right]\le 2e^{-{\lambda }^{2}m/2}.$$

$$f$$

$$(f,g):=\frac{1}{2^n}\sum _{x\in \{0,1{\}}^n}f(x)g(x)=E[f\cdot g].$$

$$||f||:=\sqrt{(f,f)}=\sqrt{E[f^2]}.$$

$${\chi }_a(x)=(-1{)}^{a^{T}x}=(-1{)}^{{\sum }_{i=1}^n{a}_i{x}_i}.$$

$$f(x)=\sum _{a\in \{0,1{\}}^n}\hat{f}(a)\cdot {\chi }_a(x)$$

$$\hat{f}(a)=E\left[f(x){\chi }_a(x)\right].$$

$$E[f^2]=\sum _{a\in \{0,1{\}}^n}{\hat{f}}^2(a)=1.$$

$$f(x_1,\cdots ,x_n)=\sum _{i=1}^s{\alpha }_i{T}_i,$$

$$T_i={\ell }_{i_1}{\ell }_{i_2}\cdots {\ell }_{i_{d^{\prime }}},$$

$$f(x_1,x_2,x_3)=x_2{x}_3+{\overline{x}}_2{x}_1+{\overline{x}}_2{\overline{x}}_1{x}_3-x_2{\overline{x}}_3-{\overline{x}}_1{\overline{x}}_2{\overline{x}}_3$$

$$\begin{array}[]{ll}x_{2}x_{3}=&\left(\frac{1-(-1)^{x_{2}}}{2}\right)\left(\frac{1-(-1)^{x_{3}}}{2}\right)\\ &\\ &=\frac{1}{4}-\frac{(-1)^{x_{2}}}{4}-\frac{(-1)^{x_{3}}}{4}+\frac{(-1)^{x_{2}+x_{3}}}{4}\\ &\\ &=\frac{1}{4}\chi_{0000}-\frac{1}{4}\chi_{0100}-\frac{1}{4}\chi_{0010}+\frac{1}{4}\chi_{0110}.\end{array}$$

$$\begin{array}[]{ll}\overline{x}_{2}x_{1}=&\left(\frac{1+(-1)^{x_{2}}}{2}\right)\left(\frac{1-(-1)^{x_{1}}}{2}\right)\\ &\\ &=\frac{1}{4}-\frac{(-1)^{x_{1}}}{4}+\frac{(-1)^{x_{2}}}{4}-\frac{(-1)^{x_{2}+x_{1}}}{4}\\ &\\ &=\frac{1}{4}\chi_{0000}-\frac{1}{4}\chi_{1000}+\frac{1}{4}\chi_{0100}-\frac{1}{4}\chi_{1100}.\end{array}$$

$$\hat{f}(a)=\frac{k}{2^d}$$

$$F_{\alpha }\triangleq {\sum }_{x\in \{0,1{\}}^{n-r}}\hat{f}(\alpha x{)}^2=E_{y,z\in \{0,1{\}}^r,x\in \{0,1{\}}^{n-r}}[f(yx)f(zx){\chi }_{\alpha }(y){\chi }_{\alpha }(z)],$$

$$F_{\alpha }=\frac{\ell }{2^{2d}}$$

$$\mathrm{Pr}\left[\left|\frac{1}{m}\sum _{i=1}^m{W}_i-F_{\alpha }\right|\ge \frac{1}{2^{2d+1}}\right]\le 2e^{-m/2^{4d+1}}<\delta .$$

$$\left|\frac{1}{m}\sum _{i=1}^m{W}_i-F_{\alpha }\right|<\frac{1}{2^{2d+1}}.$$

$$\Pr\Big{[}x_{i_{1}}=\xi_{1},\cdots,x_{i_{d}}=\xi_{d}\Big{]}=\frac{1}{2^{d}}.$$

$$\left|\operatorname*{\mathbb{E}}_{S^{\prime}}[\chi_{\alpha}(x)\Big{]}\right|\leq\lambda.$$

$$\frac{n}{{\lambda }^{2+o(1)}}$$

$$\left|\operatorname*{\mathbb{E}}_{S^{\prime}}[\chi_{\alpha}(x)\Big{]}\right|\leq\lambda.$$

$$O\left(\frac{d\log n}{{\lambda }^{2+o(1)}}\right)$$

$$E_{x\in S}[f(x{)}^2]=\sum _{a\in S}\hat{f}(a{)}^2.$$

$$L_1(f)\triangleq \sum _z|\hat{f}(z)|.$$

$$\lambda ={{{\max }_{a,b\in S}}_{a\ne b}}_{\hat{f}(a),\hat{f}(b)\ne 0}|E_{S^{\prime }}[{\chi }_{a+b}(x)]|,$$

Full text

Keywords: Decision Trees, Membership Queries, Adaptive Learning, Exact Learning

1 Introduction

Learning from membership queries, [1], has fueled the interest of many papers due to its diverse applications in various areas including group testing [2], pooling design for DNA sequencing [3], blood testing [4], functional genomics (molecular biology) [5], chemical reactions [6, 7], multi-access channel communications [2, 8], whole-genome shotgun sequencing [9], DNA physical mapping [10], game theory [11], program synthesis [12], channel leak testing, codes, VLSI testing and AIDS screening and many others as in [13] and the references therein. Generally, in domains where queries can be answered by a non-human annotator, e.g. from experiments such as laboratory tests, learning from membership query may be a promising direction for automated scientific discovery.

In this paper, we consider the problem of learning Boolean decision trees from membership queries provided that the tree is of depth at most $d$. Learning decision trees from membership queries is equivalent to solving problems in widespread areas including drug development, software update problem, bioinformatics, chemoinformatics and many others.

1.1 Applications

In the sequel we bring two examples where learning decision tree in the exact learning model might be applied to improve scientific and technological research.

Drug development. The process of drugs discovery starts by searching for lead compounds in a huge library of chemical molecules [14, 15, 16]. A compound is called active with respect to some (biological) problem if it exhibits a very high activity in a biological test or assay. For example, checking if some molecule binds with a specific protein or cancerous cell. The essential goal is to find, at a very early stage, some lead compounds that exhibit a very high activity in the assay. One of the famous methods to cope with this problem is building Quantitive Structure-Activity Relationship model or QSAR. The QSAR methodology focuses on finding a model, which correlates between activity and the compound structure. A molecule is described using a predefined set of features or descriptors selected according to chemical considerations. Decision trees are one of the standard ways used in drug research to model such relations. A membership query can be simulated by building a compound according to the specific descriptors defined by the algorithm’s query and screening it against the biological assay. Compounds that exhibit high activity against the assay are related to as positive queries, while inactive compounds are related as negative ones.

**The software update problem. ** Complicated software systems are made up of many interacting packages that must be deployed and coexist in the same context. One of the tough challenges system administrators face is to know which software upgrades are safe to apply to their particular environment without breaking the running and working system, thus causing downtime. The software upgrade problem is to determine which updates to perform without breaking the system [17]. Breaking the system might be as a result of conflicting packages, defective packages or dependent packages. Let $P$ denote the set of packages. An installation is a set of packages $I\subseteq P$. An installation can be successful depending on whether all dependencies, conflicts and defect constraints are satisfied. Let $p,q\in P$. A dependency $p\rightarrow q$ is a constraint that means any successful installation that contains the package $p$ must contain $q$. A conflict $\{p,q\}$ implies that any installation that includes both $p$ and $q$ together will fail. Finally, a package $p$ is a defect if it cannot be part of any successful installation. Consider that the dependencies, conflicts and defects are not known to the user, the purpose is to learn a model that for any installation can determine whether the installation will succeed or not. Let $P=\{p_{1},\cdots,p_{n}\}$, $R$ be the set of the dependencies, $C$ the set of conflicting packages and $D$ the set of defectives. Let $x_{1},\cdots,x_{n}$ be the variables indicating whether the package participates in the installation or not. This is equivalent to learning decision tree of depth $d$ from membership queries where a membership query is simulated by actually making the installation and getting a status feedback about it and $d=O(|C|+|R|+|D|)$. For the cases where $d<n$, learning decision trees of depth at most $d$ might be an efficient solution.

1.2 The Learning Model

In the exact learning from membership queries model [1], we have a teacher and a learner. The teacher holds a hidden target function or concept $f:\{0,1\}^{n}\rightarrow\{0,1\}$ from a fixed class of Boolean functions $C$ that is known to the learner. The target of the learner is to discover the function $f$. Given an access to the teacher, the learner can ask the teacher membership queries by sending it elements $a\in\{0,1\}^{n}$. As a response, the teacher returns an answer $f(a)\in\{0,1\}$. The main target of the learner is to learn the exact target function $f$, using a minimum number of membership queries and a small time complexity. We regard the learner as a learning algorithm and the teacher as an oracle $\mathrm{MQ}_{f}$ that answers membership queries, i.e., $\mathrm{MQ}_{f}(a)=f(a)$.

Let $C$ and $H$ be two classes of representations of Boolean functions. Let $C$ be a subset of $H$ in a sense that each function in $C$ has a representation in $H$. We say that a learning algorithm $A$ exactly learns $C$ from $H$ in time $t(A)$ with (or, and asks) $q(A)$ membership queries if for every target function $f\in C$, the algorithm $A$ runs in time at most $t(A)$, asks at most $q(A)$ membership queries and outputs a function $h\in H$ that is logically equivalent to $f$ ($f=h$). If such an algorithm exists, then we say that the class $C$ is learnable from $H$ in time $t(A)$ with $q(A)$ membership queries (or just, queries).

The learning algorithm can be deterministic or randomized, adaptive (AD) or non-adaptive (NAD). In the adaptive learning algorithm, the queries might rely on the results of previous ones. On the other hand, in the non-adaptive algorithm, the queries are independent and must not rely on previous results. Therefore, all the queries can be asked in a single parallel step. We say that an adaptive algorithm is an $r$-round adaptive ($r$-RAD) if it runs in $r$ stages where each stage is non-adaptive. That is, the queries may depend on the answers of the queries in previous stages, but are independent of the answers of the current stage queries.

A randomized algorithm is called a Monte Carlo (MC) algorithm if it uses coin tosses and its running time is finite, while its output might be incorrect with a probability of at most $\delta$, that is specified as an input to the algorithm. All of the randomized algorithms regarded herein are Monte Carlo algorithms.

Let $\mathrm{DT}_{d}$ denote the class of decision trees of depth at most $d$. We say that the class $\mathrm{DT}_{d}$ is polynomial time learnable if there is an algorithm that learns $\mathrm{DT}_{d}$ in $poly(2^{d},n)$ time and asks $poly(2^{d},\log n)$ membership queries. We cannot expect better time or query complexity since just asking one membership query takes $O(n)$ time. As we will see below, the lower bound for the number of queries is $\Omega(2^{d}\log n)$.

Along the discussion, we will use the notation $\mathrm{OPT}(\mathrm{DT}_{d})$ to indicate the optimal number of membership queries needed to learn the class $\mathrm{DT}_{d}$ with unlimited computational power. More precisely, we use the notation $\mathrm{OPT}_{\mathrm{AD}}(\mathrm{DT}_{d})$ ($\mathrm{OPT}_{\mathrm{NAD}}(\mathrm{DT}_{d})$, $\mathrm{OPT}_{\mathrm{R},\mathrm{AD}}(\mathrm{DT}_{d})$, $\mathrm{OPT}_{\mathrm{R},\mathrm{NAD}}(\mathrm{DT}_{d})$) to indicate the optimal number of membership queries needed by a deterministic adaptive (resp., deterministic non-adaptive, randomized adaptive with with failure probability of $\delta=1/2$, randomized non-adaptive with failure probability of $\delta=1/2$) algorithm to exact-learn the class $\mathrm{DT}_{d}$ from membership queries.

1.3 Known Results

In the following, we give a brief summary of some of the known results of the learnability of the class $\mathrm{DT}_{d}$. In [18, 13], it is shown that111$\tilde{O}(T)$ is $O(poly(d)\cdot T)$. We will use this notation throughout this paper. The exact upper bound is $O(d2^{2d}\log n)$.:

Theorem 1**.**

[TABLE]

and

[TABLE]

Achieving an upper bound better than $\tilde{O}(2^{2d})\log n$ is an open problem. Table 1 summarizes some known results on learning decision trees. Most of the algorithms for learning decision trees relate to the target function using the Discrete Fourier Transform representation and try to learn its non-zero Fourier coefficients. Vitaly Feldman  [19] introduced the first randomized non-adaptive algorithm that finds the heavy Fourier coefficients. Applying his algorithm to the class $\mathrm{DT}_{d}$ gives a non-adaptive learning algorithm with query complexity $\tilde{O}(2^{2d})\log^{2}n$. See [18] for the algorithm. Applying the reduction from [13] to the algorithm of Feldman gives a non-adaptive learning algorithm with query complexity $\tilde{O}(2^{2d})\log n$. We want to draw the reader’s attention that any result for the non-adaptive case implies the same result for the adaptive case. This is result (1) in Table 1. Kushilevitz and Mansour [20], and Jackson  [21], gave a randomized adaptive algorithm for learning decision trees known as KM algorithm. Kushilevitz and Mansour then use the technique of $\epsilon$-biased distribution to derandomize the algorithm. A rough estimation of the query complexity of their algorithm for the class of $\mathrm{DT}_{d}$ was first done in [18]. In Appendix A, we show that, with the best derandomization results exist today [22], the query complexity of their deterministic algorithm is $2^{18d+o(d)}\log n$. This is result (2) in Table 1.

1.4 Our Results

In this paper we give two new algorithms. The first algorithm is a randomized adaptive algorithm (two rounds) that asks $\tilde{O}(2^{2d})+2^{d}\log n$ membership queries. In particular, this shows that $\mathrm{OPT}_{\mathrm{R},\mathrm{AD}}(\mathrm{DT}_{d})\leq\tilde{O}(2^{2d})+2^{d}\log n$. The second algorithm is a deterministic adaptive algorithm that asks $2^{5.83d}+2^{2d+o(d)}\log n$. See Table 1.

For the randomized adaptive algorithm, we give a new reduction that changes a randomized $r$-round algorithm $A$ that learns a class $C$ with $Q(n)$ membership queries to a randomized $(r+1)$-round algorithm that learns $C$ with $Q(8V^{2})+v\log n$ queries, where $V$ is an upper bound on the number of relevant variables of the functions in $C$ and $v$ is the number of relevant variables of the target function. The class $C$ must be closed under variable projection. We apply this reduction to the randomized nonadaptive (one round) algorithm of Feldman for $\mathrm{DT}_{d}$. In Feldman’s algorithm we have $Q(n)=\tilde{O}(2^{2d})\log n$. Since, for $\mathrm{DT}_{d}$, $V,v\leq 2^{d}$, we get a two-round algorithm that asks $\tilde{O}(2^{2d})+2^{d}\log n$ queries.

In our reduction, we randomly uniformly project the variables $(x_{1},\ldots,x_{n})$ of the target function $f$ into $m=8V^{2}$ new variables $(y_{1},\ldots,y_{m})$. Since the number of the relevant variables of the target function is at most $V$, with high probability all the variables in the target function are projected to distinct variables. Let $g(y)$ be the projected function. We run the algorithm $A$ to learn $g(y)$. Obviously, queries to $g$ can be simulated by queries to $f$. This takes $Q(8V^{2})$ queries. Algorithm $A$ returns a function $h(y)$ equivalent to $g$. Then, we show, without asking any more membership queries, how to find the relevant variables of $g(y)$ and how to find witnesses for each relevant variable. That is, for each relevant variable $y_{i}$ in $g$, we find two assignments $a^{(i)}$ and $b^{(i)}$ that differ only in $y_{i}$ such that $g(a^{(i)})\not=g(b^{(i)})$. Moreover, we show that for each relevant variable $y_{i}$ in $g$, one can find the corresponding relevant variable in $f$ by running the folklore nonadaptive algorithm of group testing for finding one defective item combined with the witnesses. Each relevant variable can be found with $\log n$ queries. This requires $v\log n$ queries in the second round.

For the deterministic adaptive algorithm, we first find the relevant variables. Then, we learn the decision tree over the relevant variables using the multivariate polynomial representation, [24], with the derandomization studied in [25]. To find the relevant variables, we construct a combinatorial block design, that we will call an $(n,d)$-universal disjoint set, in polynomial time. A set $S\subseteq\{0,1,z\}^{n}$ is called an $(n,d)$-universal disjoint set if for every $1\leq i_{1}<\cdots<i_{d}\leq n$, every $\xi_{1},\ldots,\xi_{d}\in\{0,1\}^{n}$ and every $1\leq j\leq d$ there is an assignment $a\in S$ such that $a_{i_{k}}=\xi_{k}$ for all $k\not=j$ and $a_{i_{j}}=z$. We regard $z$ as a variable. The algorithm finds all the relevant variables in the target function as follows. To find the relevant variables, the algorithm starts with constructing an $(n,2d+1)$-universal disjoint set $S$. For any assignment $a\in S$, let $a[x]$ be the assignment $a$ where each entry $i$ that is equal to $z$ is replaced with $x_{i}$. We show that for every decision tree $f$ and every relevant variable $x_{i}$ in $f$ there is an assignment $a\in S$ such that $f(a[x])$ is equal to either $x_{i}$ or $\bar{x}_{i}$. Therefore, when $f(a[1^{n}]|_{x_{i}\leftarrow 1})\not=f(a[0^{n}]|_{x_{i}\leftarrow 0})$, we can run the algorithm that learns one literal to learn a relevant variable in $f$. To learn $f(a[x])$, we need $O(\log n)$ queries. This gives all the relevant variables of $f$ in $|S|+O(v\log n)$ queries where $v$ is the number of relevant variables in $f$.

In addition, in this paper we give a polynomial time algorithm that constructs an $(n,2d+1)$-universal disjoint set $S$ of size $|S|=2^{2d+o(d)}\log n$. This size is optimal up to $2^{o(d)}$. Hence, the total number of queries for the first stage is $2^{2d+o(d)}\log n$.

In the second stage, the algorithm learns the target function over the relevant variables that are found in the first stage. We start by showing that $\mathrm{DT}_{d}$ can be represented as $\mathrm{MP}_{d,3^{d}}$; multivariate polynomial over the field $F_{2}$ with monomials of size (number of variables in the monomial) at most $d$ and at most $3^{d}$ monomials. To learn $\mathrm{MP}_{d,s}$, $s=3^{d}$, over $N\leq 2^{d}$ variables, we first show how to learn one monomial. In order to (deterministically) learn one monomial, we use the deterministic zero test procedure Zero-Test of $\mathrm{MP}_{d,s}$ from [25] that asks $Z=2^{2.66d}s\log^{2}N$ queries. The learning algorithm of $\mathrm{MP}_{d,s}$ from [24] with this procedure gives a deterministic learning algorithm that asks $O(ZNs)$ queries. This solves the problem with query complexity $\tilde{O}(2^{6.83d})$ (for learning $\mathrm{DT}_{d}$ with $N\leq 2^{d}$ variables). In this paper, we give an algorithm that asks $O(Zs\cdot\log N)$ queries that gives query complexity $\tilde{O}(2^{5.83d})$.

To find one monomial, our algorithm uses $(N,d)$-sparse all one set $D\subseteq\{0,1\}^{N}$. An $(N,d)$-sparse all one set is a set $D\subseteq\{0,1\}^{N}$ where for every $1\leq i_{1}<\cdots<i_{d}\leq n$ there is an assignment $a\in D$ such that $a_{i_{k}}=1$ for all $k=1,\ldots,d$ and the weight of each $a\in D$ is at most $N(1-1/2d)$. We show how to construct such set in polynomial time. Thereafter, we show that for every $f\in\mathrm{DT}_{d}$, $f\not=0$, there is an assignment $a\in D$ such that $f(a*x)\not=0$ where $a*x=(a_{1}x_{1},\ldots,a_{N}x_{N})$. To find such an assignment, we use the procedure Zero-Test to zero test each $f(a*x)$, for all $a\in D$. Notice that the weight of $a$ is at most $(1-1/2d)N$ and therefore, the function $f(a*x)$ contains at most $(1-1/2d)N$ variables. Thus, if we recursively use this procedure, after at most $2d\ln N$ iterations, we get a monomial $M$ in $f$. Therefore, to find one monomial we need at most $2^{2.66d}s2d\ln N$ queries.

Assuming that we have found $t$ monomials $M_{1},\ldots,M_{t}$ of $f$, we can learn a new one by learning a monomial of $f^{\prime}=f+M_{1}+M_{2}+\cdots+M_{t}$. The function $f^{\prime}$ is the function $f$ without the monomials $M_{1},\ldots,M_{t}$. This is because $M+M=0$ in the binary field. Obviously, we can simulate queries for $f^{\prime}$ with queries for $f$. Since the number of monomials in $f$ is at most $s=3^{d}$, the number of queries in the second stage is at most $2^{2.66d}s^{2}2d\ln N=\tilde{O}(2^{5.83d})$.

2 Preliminaries

Let $n$ be an integer. We denote $[n]=\{1,\cdots,n\}$. Consider the set of assignments, also called membership queries or queries, $\{0,1\}^{n}$. A function $f$ is called a Boolean function on $n$ variables $\{x_{1},\cdots,x_{n}\}$ if $f:\{0,1\}^{n}\rightarrow\{0,1\}$. We denote by $1^{n}$ ($0^{n}$) the all one (zero) assignment. For an assignment $a$, the $i$-th entry in $a$ is denoted by $a_{i}$. For two assignments $a,b\in\{0,1\}^{n}$, the assignment $a+b$ denotes the bitwise xor of the assignments $a$ and $b$. For an assignment $a\in\{0,1\}^{n}$, $i\in[n]$ and $\xi\in\{0,1\}$, we denote by $a|_{x_{i}\leftarrow\xi}$ the assignment $(a_{1},\cdots,a_{i-1},\xi,a_{i+1},\cdots,a_{n})$. We say that the variable $x_{i}$ is relevant in $f$ if there is an assignment $a=(a_{1},\cdots,a_{n})$ such that $f(a|_{x_{i}\leftarrow 0})\neq f(a|_{x_{i}\leftarrow 1})$. We call such assignment a witness assignment for $x_{i}$. A Boolean decision tree is defined as follows:

Definition 1**.**

Let $\mathrm{DT}$ denote the set of all Boolean decision tree formulae. Then, 1) $f\equiv 0\in\mathrm{DT}$ and $f\equiv 1\in\mathrm{DT}$ and 2) if $f_{0},f_{1}\in\mathrm{DT}$, then for all $x_{i}$, $f:=\overline{x}_{i}f_{0}+x_{i}f_{1}\in\mathrm{DT}$.

An immediate observation from the construction of $f$, as defined by step 2 in Definition 1, is that the terms of $f$ are disjoint i.e. there is at most one non-zero term for the same assignment. Consequently, the “$+$” operation can be replaced with the logic “$\vee$”. Hence, we can relate to the function $f$ as a disjunction of pairwise disjoint terms.

Every decision tree $f$ can be modelled as a binary tree denoted by $T(f)$ or shortly $T$ if $f$ is known from the context. Let $\cal{V}$ denote the vertices set of $T$. If $f\equiv 0$ or $f\equiv 1$, then $T$ is a node labelled with [math] or $1$ respectively. Otherwise, by Definition 1, there are some $x_{i}$ and $f_{0},f_{1}\in\mathrm{DT}$ such that $f=\overline{x}_{i}f_{0}+x_{i}f_{1}$. Then, $T$ has a root node $r\in\cal{V}$ labelled with $x_{i}$ and two outgoing edges. The left edge is labelled by zero and points to the root node of $T(f_{0})$. The right edge is labelled by $1$ and points to the root node of $T(f_{1})$. The size of a decision tree $T$ is the number of its leaves. For any tree $T$ we define $depth(T)$ to be the number of the edges of the longest path from the root to a leaf in $T$. Let $T_{f}$ denote all the decision tree models of the Boolean function $f$. Let $d$ be the depth of the decision tree model $T(f)$. We say that $T(f)$ is a minimal model for $f$ if $d=\mathop{\min}_{T\in T_{f}}depth(T)$. The depth of a decision tree $f$ is the depth of its minimal model. We denote by $\mathrm{DT}_{d}$ the class of all decision trees of depth at most $d$.

Given an assignment $a\in\{0,1\}^{n}$, the decision tree model $T(f)$ defines a computation. The computation traverses a path from the root to a leaf and assigns values to the variables indicated by the inner nodes in the following way. The computation starts from the root node of the tree $T(f)$. When the computation arrives to an inner node labeled by $x_{i}$, it assigns the value of the input $a_{i}$ to $x_{i}$. If $a_{i}=1$, then the computation proceeds to the right child of the current node. Otherwise, it proceeds to the left child. The computation terminates when a leaf $u$ is reached. The value of the computation is the value indicated by the leaf $u$.

Lemma 2**.**

Let $f_{1},f_{2}\in\mathrm{DT}_{d}$, then for any boolean function $g(y_{1},y_{2})$ we have $g(f_{1},f_{2})\in\mathrm{DT}_{2d}$.

Proof.

We construct the following tree denoted by $[f_{1},f_{2}]_{g}$. We replace each leaf $v$ of $f_{1}$ by the tree $f_{2}$. Denote by $w_{u,v}$ the leaf $u$ of the tree $f_{2}$ that replaces $v$ in $f_{1}$. The label of $w_{u,v}$ is $g(\ell(v),\ell(u))$, where $\ell(v)$ is the label of $v$ in $f_{1}$ and $\ell(u)$ is the label of $u$ in $f_{2}$. Therefore, any assignment $a$ that reaches the leaf $v$ in $f_{1}$ and $u$ in $f_{2}$ satisfies $g(f_{1},f_{2})(a)=g(f_{1}(a),f_{2}(a))=g(\ell(v),\ell(u))$. On the other hand, this assignment reaches the leaf $w_{u,v}$ in $[f_{1},f_{2}]_{g}$ which has the label $g(\ell(v),\ell(u))$. The result follows since $[f_{1},f_{2}]_{g}\in\mathrm{DT}_{2d}$. ∎

2.1 Multivariate Polynomial Representation of Decision Trees

A monotone term or monomial is a product of variables. The size of a monomial is the number of the variables in it. A multivariate polynomial is a linear combination (over the binary field $F_{2}$) of distinct monomials. The size of a multivariate polynomial is the number of monomials in it. It is well known that multivariate polynomials have a unique representation so the size is well defined. Let $\mathrm{MP}_{d,s}$ be the set of Boolean multivariate polynomials over the binary field $F_{2}$ with at most $s$ monomials each of size at most $d$. We say that a set $Z\subseteq\{0,1\}^{n}$ is a zero test for $\mathrm{MP}_{d,s}$ if for every $f\in\mathrm{MP}_{d,s}$ and $f\neq 0$, there is an assignment $a\in Z$ such that $f(a)\not=0$. In [25](Lemma 72), it is proved:

Lemma 3**.**

There is a zero test set $Z\subseteq\{0,1\}^{n}$ for $\mathrm{MP}_{d,s}$ of size $2^{2.66d}s\log^{2}n$ that can be constructed in polynomial time $poly(2^{d},n)$.

Lemma 4**.**

Let $f\in\mathrm{DT}_{d}$. Then, $f$ is equivalent to some $h\in\mathrm{MP}_{d,3^{d}}$.

Proof.

The proof is by induction on $d$. For $d=0$, the decision tree is either [math] or $1$. The size is at most $1\leq 3^{0}$. For $d=1$, the decision tree is either $x_{i}$ or $x_{i}+1$. The size is at most $2\leq 3^{1}$. Suppose the hypothesis is true for $d-1$. Let $f$ be a decision tree of depth $d$ and let $x_{i}$ be the label of its root. Then, $f=x_{i}f_{1}+(x_{i}+1)f_{2}$ where $f_{1}$ and $f_{2}$ are trees of depth $d-1$. By the induction hypothesis, $f_{1},f_{2}$ are equivalent to $h_{1},h_{2}\in\mathrm{MP}_{d-1,3^{d-1}}$. Since $f=x_{i}f_{1}+x_{i}f_{2}+f_{2}=x_{i}h_{1}+x_{i}h_{2}+h_{2}\in\mathrm{MP}_{d,3^{d}}$, the result follows. ∎

2.2 Block Designs

In this section we define two block designs that will be used in the deterministic algorithm for learning decision trees.

A set $S\subseteq\{0,1,z\}^{n}$ is called an $(n,d)$-universal disjoint set if for every $1\leq i_{1}<\cdots<i_{d}\leq n$, every $\xi_{1},\ldots,\xi_{d}\in\{0,1\}$ and every $1\leq j\leq d$, there is an assignment $a\in S$ such that $a_{i_{k}}=\xi_{k}$ for all $k\not=j$ and $a_{i_{j}}=z$. We regard $z$ as a variable.

Let $H$ be a family of functions $h:[n]\to[q]$. For $d\leq q$, we say that $H$ is $(n,q,d)$-perfect hash family if for every subset $S\subseteq[n]$ of size $|S|=d$ there is a hash function $h\in H$ such that $|h(S)|=d$.

Bshouty  [26] shows that:

Lemma 5**.**

Let $q$ be a power of prime. If $q>2d^{2}$, then there is an $(n,q,d)$-perfect hash family of size

[TABLE]

that can be constructed in linear time.

The following lemma is proved in [27]:

Lemma 6**.**

An $(8d^{2},d)$-universal set of size $2^{d+O(\log^{2}d)}$ can be constructed in $poly(2^{d},n)$ time.

We now prove:

Lemma 7**.**

There is an $(n,d)$-universal disjoint set $S\subseteq\{0,1,z\}^{n}$ of size $2^{d+O(\log^{2}d)}$ $\log n$ that can be constructed in polynomial time.

Proof.

Consider the $(8d^{2},d)$-universal set $U$ of size $2^{d+O(\log^{2}d)}$ from Lemma 6. Define a set $W\subseteq\{0,1,z\}^{t}$, for $t=8d^{2}$ as follows. For each $a\in U$, add $t$ vectors $a^{(1)},\ldots,a^{(t)}$ to $W$ where $a^{(i)}=a|_{x_{i}\leftarrow z}=(a^{(i)}_{1},\ldots,a^{(i)}_{i-1},z,a^{(i)}_{i+1},\ldots,a^{(i)}_{t})$. Obviously, $W$ is $(8d^{2},d)$-universal disjoint set of size $\ell=2^{d+O(\log^{2}d)}$. Let $w^{(1)},\ldots,w^{(\ell)}$ be the vectors in $W$. Let $4d^{2}<q=2^{r}\leq 8d^{2}$, and consider an $(n,q,d)$-perfect hash family $H$. By Lemma 5, $H$ is of size $O(d^{2}\log n)$, and can be constructed in linear time. We use $W$ and $H$ to construct an $(n,d)$-universal disjoint set $S$ as follows. For every $h\in H$ and every $w\in W$, we add the assignment $[w,h]=(w_{h(1)},\ldots,w_{h(n)})$ to $S$.

First, the size of $S$ is $2^{d+O(\log^{2}d)}\log n$. Moreover, consider $1\leq i_{1}<\cdots<i_{d}\leq n$ and any $\xi_{1},\ldots,\xi_{d}\in\{0,1\}$. Since $H$ is an $(n,q,d)$-perfect hash family, there is a hash function $h\in H$ such that $r_{1}=h(i_{1}),\ldots,r_{d}=h(i_{d})\in[q]$ are distinct. Since $W$ is an $(8d^{2},d)$-universal disjoint set and $q\leq 8d^{2}$ for every $1\leq j\leq d$, there is an assignment $w\in S$ such that $w_{r_{k}}=\xi_{k}$ for all $k\not=j$ and $w_{r_{j}}=z$. Therefore, $[w,h]_{i_{k}}=w_{h(i_{k})}=w_{r_{k}}=\xi_{k}$ and $[w,h]_{i_{j}}=z$. Thus, $S$ is an $(n,d)$-universal disjoint set. ∎

An $(n,d)$-sparse all one set $D\subseteq\{0,1\}^{n}$ is a set such that for every $a\in D$, $wt(a):=|\{i|a_{i}=1\}|\leq n(1-1/(2d))$, and for every $1\leq i_{1}<\cdots<i_{d}\leq n$ there is an assignment $a\in D$ such that $a_{i_{k}}=1$ for all $k\in[d]$.

Lemma 8**.**

There is an $(n,d)$-sparse all one set $D\subseteq\{0,1\}^{n}$ of size $d+1$ that can be constructed in polynomial time.

Proof.

Consider the $d+1$ vectors $a^{(j)}\in\{0,1\}^{n}$, $j=0,1,\ldots,d$ where $a^{(j)}_{i}=0$ if $i\mod(d+1)=j$ and $a^{(j)}_{i}=1$ otherwise. Consider any $1\leq i_{1}<i_{2}<\cdots<i_{d}\leq n$. Let $j_{t}=i_{t}$ $mod(d+1)$ for $t\in[d]$. Therefore, $j_{1},\cdots,j_{d}$ take at most $d$ distinct values from $S=\{0,1,\cdots,d\}$. As a result, there is $k\in S$ such that $j_{\ell}\not=k$ for all $\ell=1,\ldots,d$. Then, obviously by the construction of $D$, $a^{(k)}_{i_{1}}=\cdots=a^{(k)}_{i_{d}}=1$. Moreover, for each $j$, $wt(a^{(j)})\leq n-\lceil n/(d+1)\rceil\leq n(1-1/(2d)).$ ∎

3 Randomized Adaptive Learning

In this section we give a randomized two-round algorithm that learns $\mathrm{DT}_{d}$ in polynomial time with $\tilde{O}(2^{2d})+2^{d}\log n$ queries. First, we give a reduction that changes a randomized $r$-round learning algorithm to a randomized $(r+1)$-round learning algorithm while reducing the number of queries for classes with small number of relevant variables. We use this reduction for $\mathrm{DT}_{d}$ with Feldman’s algorithm to get the above result.

An $m$-variable projection is a function $P:\{x_{1},\ldots,x_{n}\}\to\{y_{1},\ldots,y_{m}\}$ where $\{y_{1},\ldots,y_{m}\}$ are new distinct variables. For $x=(x_{1},\ldots,x_{n})$ we write $P(x)=(P(x_{1}),\ldots,$ $P(x_{n}))$, where for an assignment $a=(a_{1},\cdots,a_{n})$, $P(x_{i})(a)=a_{j}$ if $P(x_{i})=y_{j}$. Let $C$ be a class of Boolean functions. We say that $C$ is closed under variable projection if for every $f\in C$, every integer $m$ and every $m$-variable projection $P$ we have $f(P(x))\in C$. We say that $C$ is $t(n)$-verifiable (with success probability at least $1-\delta$) if for every $f\in C$ with $n$ variables one can find, in time $t(n)$ (with probability at least $1-\delta$), the relevant variables of $f$, and can derive a witness for each relevant variable $x_{i}$, i.e. an assignment $a^{(i)}\in\{0,1\}^{n}$ such that $f(a^{(i)}|_{x_{i}\leftarrow 1})\not=f(a^{(i)}|_{x_{i}\leftarrow 0})$. We denote by $V(f)$ the number of relevant variables of $f$. Let $V(C)=\max_{f\in C}V(f)$. The reduction is described in Figure 1. Now we can prove:

Theorem 9**.**

Let $C$ be a class of functions that is closed under variable projection. $m=8V(C)^{2}$. Let $H$ be a $t(n)$-verifiable class with success probability at least $15/16$. Let ${\cal A}_{n}$ be a randomized $r$-round algorithm that learns the class $C$ from $H$ in time $T(n)$ with $Q(n)$ membership queries and success probability at least $15/16$. Then, there is a randomized $(r+1)$-round algorithm that learns $C$ from $H$ in time $O(T(m)+nQ(m)+t(m)+V(f)n\log n)$ with $Q(m)+V(f)(\log n+1)$ queries and success probability at least $3/4$, where $f$ is the target function.

Proof.

Let $P$ be a random uniform $m$-variable projection (step 1 in Figure 1). The algorithm runs ${\cal A}_{m}$ to learn $g(y)=f(P(x))$ (step 2). Since $C$ is closed under variable projection, $g(y)=f(P(x))\in C$. Therefore, with probability at least $15/16$, ${\cal A}_{m}$ returns a function $h(y)$ equivalent to $g(y)$. In Lemma 12, we show that, with probability at least $15/16$, the relevant variables of $f$ are mapped by $P$ to different $y_{i}$. Therefore, with probability at least $7/8$, $h(y)$ is equivalent to $g(y)$ and the relevant variables of $f$ are mapped by $P$ to different $y_{i}$. We now proceed assuming such events are true.

Since $g(y)=h(y)\in H$ and $H$ is $t(n)$-verifiable with success probability at least $15/16$, one can find the relevant variables of $g$ and witnesses for each relevant variable in time $t(m)$ and success probability at least $15/16$. That is, for each relevant variable $y_{i}$ in $g$, the procedure Find-Witness in step (3), with probability at least $15/16$, finds an assignment $a^{(i)}\in\{0,1\}^{m}$ and $\xi_{i}\in\{0,1\}$ such that $g(a^{(i)}|_{y_{i}\leftarrow\xi_{i}})=0$ and $g(a^{(i)}|_{y_{i}\leftarrow\bar{\xi}_{i}})=1$. The procedure Find-Witness returns a set $Y$ of all the relevant variables of $g(y)$ and a set $A$ containing the corresponding witnesses of the relevant variables in $g$.

Notice that, for all $y_{\ell}\in Y$, $g(a^{(\ell)}|_{y_{\ell}\leftarrow\xi_{\ell}+z})=z$, $z\in\{0,1\}$. Consider some $y_{\ell}\in Y$ and let $X_{\ell}:=P^{-1}(y_{\ell})=\{x_{j}\ |\ P(x_{j})=y_{\ell}\}$. That is, all the variables $x_{j}$ that are mapped to $y_{\ell}$ by $P$. Notice that, if $X$ is the set of relevant variables of $f$, then for each $y_{\ell}\in Y$, the set $X\cap P^{-1}(y_{\ell})$ contains exactly one variable that maps to $y_{\ell}$. To find this variable, suppose $P(x)=(y_{j_{1}},\ldots,y_{j_{n}})$. Then $g(a^{(\ell)})=f(a^{(\ell)}_{j_{1}},\ldots,a^{(\ell)}_{j_{n}})$. Since $f$ has exactly one relevant variable in $X_{\ell}$ and $g(a^{(\ell)}|_{y_{\ell}\leftarrow\xi_{\ell}+z})=z$, we have $f(\beta^{(\ell)})=g(a^{(\ell)}|_{y_{\ell}\leftarrow\xi_{\ell}+z})=z$ where $\beta^{(\ell)}=(\beta_{1}^{(\ell)},\ldots,\beta_{n}^{(\ell)})$, $\beta_{i}^{(\ell)}=\xi_{\ell}+z$ if $x_{i}\in X_{\ell}$ and $\beta_{i}^{(\ell)}=a^{(\ell)}_{j_{i}}$ if $x_{i}\notin X_{\ell}$. Therefore, $f(b^{(\ell)})=x_{t}$ where $b^{(\ell)}_{i}=\xi_{\ell}+x_{i}$ if $x_{i}\in X_{\ell}$ and $b^{(\ell)}_{i}=a^{(\ell)}_{j_{i}}$ if $x_{i}\notin X_{\ell}$. Therefore, by learning $f(b^{(\ell)})$ we find the relevant variable in $X_{\ell}$. These are steps 4.1-4.3. In Lemma 13, we show how to learn one variable non-adaptively with at most $\log n+1$ queries. This is the procedure Learn-Variable in step 4.4. It returns $k_{\ell}$ where $x_{k_{\ell}}$ is the variable that corresponds to $y_{\ell}$. Finally, we replace each relevant variable $y_{\ell}\in Y$ in $g$ with $x_{k_{\ell}}$ and get the target function (Step 5).

The success probability of the algorithm is at least $13/16>3/4$. The time complexity in step 2 is $O(T(m)+nQ(m))$, in step 3 is $t(m)$, and in step 4.4, by Lemma 13, is $O(V(f)n\log n)$. This gives $O(T(m)+nQ(m)+t(m)+V(f)n\log n)$ time complexity. The query complexity is $Q(m)$ in step 2 and $V(f)\lceil\log n\rceil$ in step 4.4. The number of rounds is $r$ rounds for running ${\cal A}_{m}$ and one round for finding the relevant variables in step 4. ∎

In Lemma 11, we show that if the class $C$ is closed under variable projection and is learnable from $H$ in time $T(n)$ and $Q(n)$ queries, then it is verifiable in time $t(n)=O((T(n)+Q(n)\tau(n))n\log n)$ where $\tau(n)$ is the time to compute a function in $H$. In particular, the result of Theorem 9 is true in time $O((T(m)+Q(m)\tau(m))m\log m+nQ(m)+V(f)n\log n).$ In particular, we have:

Corollary 10**.**

The class $\mathrm{DT}_{d}$ is $2$-round learnable with a randomized algorithm that runs in polynomial time and asks $\tilde{O}(2^{2d})+2^{d}\log n$ queries.

Proof.

We use Feldman’s non-adaptive (one round) randomized algorithm that asks $Q(n)=O(d^{3}2^{2d}\log^{2}n)$ queries and runs in $T(n)=poly(2^{d},n)$ time. In Lemma 25 in Appendix A, we show that the output of Feldman’s algorithm is verifiable in time $O(2^{2d}n)$. For $\mathrm{DT}_{d}$, we have $V(C)=2^{d}$. By Theorem 9, we get a two round randomized algorithm that runs in time $poly(2^{d},n)$ and asks $O(d^{5}2^{2d}+2^{d}\log n)$ queries. ∎

3.1 Proof of the Lemmas

In this subsection we prove the lemmas used in the previous section.

Lemma 11**.**

Let $C$ be a class of functions that is closed under variable projection. Let ${\cal A}_{n}$ be a randomized learning algorithm that learns $C$ from $H$ in time $T(n)$ with $Q(n)$ queries and success probability at least $15/16$. Let $g(y_{1},\ldots,y_{m})\in C$ be a computable function in time $\tau(m)$. Then, the relevant variables of $g$ and a witness $a^{(i)}$ for each relevant variable $y_{i}$ in $g$ can be found in time $O((T(m)+Q(m)\tau(m))m\log m)$ and success probability at least $15/16$.

Proof.

We first give an algorithm that for any two functions $g_{1},g_{2}\in C$ in $m$ variables, with success probability at least $1-1/(16m)$, verifies whether they are equivalent, in time $O((T(m)+Q(m)\tau(m))\log m)$. In case they are not equivalent, the algorithm returns an assignment $a$ such that $g_{1}(a)\not=g_{2}(a)$. We run the learning algorithm ${\cal A}_{m}$. For each query $a$, we check whether $g_{1}(a)=g_{2}(a)$. If not, then we are done. Otherwise, stop and return $g_{1}(a)(=g_{2}(a))$. If for all the membership queries both functions are equal and the algorithm returns $h$, then $\Pr[g_{1}\not=g_{2}]\leq\Pr[g_{1}\not=h]+\Pr[g_{2}\not=h]\leq 1/8$. Running the algorithm $O(\log m)$ times gives failure probability $1/(16m)$. For each $2\leq i\leq m$, define $g^{(i)}(y)=g(y_{y_{i}\leftarrow y_{1}})$ and $g^{(1)}(y)=g(y_{y_{1}\leftarrow y_{2}})$. Since $C$ is closed under variable projection, we have $g^{(i)}\in C$. Notice that $y_{i}$ is a relevant variable if and only if $g(y)\not=g^{(i)}(y)$. The above algorithm can check whether $g=g^{(i)}$ for all $i$ with success probability at least $15/16$. In case, for some $i$, we get an assignment $a$ such that $g^{(i)}(a)\not=g(a)$ then (assume $i>1$),

[TABLE]

This can happen only when $a_{1}\not=a_{i}$ and then the assignment $a$ is a witness for $y_{i}$. ∎

Lemma 12**.**

Let $1\leq i_{1}\leq\cdots\leq i_{k}\leq n$. For a random $ck^{2}$-variable projection $P$ with probability at least $1-1/(2c)$, $P(x_{i_{1}}),\ldots,P(x_{i_{k}})$ are distinct.

Proof.

The probability that $P(x_{i_{j}})$ is equal to $P(x_{i_{k}})$ for some $j\not=k$ is $1/(ck^{2})$. Therefore, by the union bound, the probability that $P(x_{i_{1}}),\ldots,P(x_{i_{k}})$ are distinct is at least $1-{k\choose 2}/(ck^{2})\geq 1-1/(2c)$. ∎

In the following, we give a non-adaptive algorithm that learns one variable (the procedure Learn-Variable). This result is a folklore result in group testing.

Lemma 13**.**

There is a non-adaptive learning algorithm that asks $\lceil\log n\rceil$ queries and learns the class $C=\{x_{1},\ldots,x_{n}\}$.

Proof.

Denote by $b_{i}(j)$ the $i$-th bit of the binary representation of $j$. Define the queries $a^{(i)}=(b_{i}(0),\ldots,b_{i}(n-1))$, $i=1,\ldots,\lceil\log n\rceil$. If the target function is $x_{\ell}$, then the vector of answers to the queries is the binary representation of $\ell-1$. ∎

4 Deterministic Adaptive Learning

In the following, we give a deterministic algorithm for learning $\mathrm{DT}_{d}$. In subsection 4.1, we show that there is a polynomial time deterministic algorithm that finds all the relevant variables in $2^{2d+O(\log^{2}d)}\log n$ queries. In subsection 4.2, we show that the class $\mathrm{MP}_{d,s}$, defined over $N$ variables, can be learned in a deterministic polynomial time with $2^{2.66d}s^{2}\log^{3}N$ queries.

Theorem 14**.**

The class $\mathrm{DT}_{d}$ is deterministic polynomial time learnable from $\mathrm{MP}_{d,3^{d}}$ with $2^{5.83d}+2^{2d+O(\log^{2}d)}\log n$ queries.

Proof.

By Lemma 19, we can find the relevant variables in $2^{2d+O(\log^{2}d)}\log n$ queries. The number of relevant variables $N$ is bounded by $N\leq 2^{d}$. Since, by Lemma 4, each decision tree of depth at most $d$ has a representation in $\mathrm{MP}_{d,3^{d}}$, we can learn it as a multivariate polynomial over the $N$ variables. By Theorem 23, this takes $2^{2.66d}s^{2}\log^{3}N=d^{3}2^{2.66d}3^{2d}\leq 2^{5.83d}$ queries. ∎

4.1 Finding the Relevant Variables

In this section we give an algorithm that asks $2^{2d+O(\log^{2}d)}\log n$ queries and finds all the relevant variables of $f\in\mathrm{DT}_{d}$. We first give a folklore procedure Binary-Learn-Variable that takes two assignments $a,b\in\{0,1\}^{n}$ such that $f(a)\not=f(b)$ and finds a relevant variable $x_{j}$ of $f$ such that $a_{j}+b_{j}=1$ in $\lceil\log n\rceil$ queries. The procedure defines an assignment $c$ that results from flipping $\lfloor wt(a+b)/2\rfloor$ entries in $a$ that differ from $b$. Then, it finds $f(c)$ with a membership query. We either have $f(a)\not=f(c)$ or $f(b)\not=f(c)$ because, otherwise, $f(a)=f(b)$ and then we get a contradiction. In the first case, $a$ differs from $c$ in $\lfloor wt(a+b)/2\rfloor$ entries, while in the second case, $c$ differs from $b$ in $\lceil wt(a+b)/2\rceil$ entries. Then, we recursively do the above until, at the $t$th stage, both assignments differ in $s=\lceil wt(a+b)/2^{t}\rceil=1$ entries. Therefore, the variable that corresponds to this entry is relevant in $f$. The number of iterations until $s=1$ is at most $\lceil\log n\rceil$.

We remind the reader that for $a\in\{0,1,z\}^{n}$, $a[x]$ is the assignment $a$ where each entry $i$ that is equal to $z$ is replaced with $x_{i}$. For example, for $a=(1,0,z,1,z)$, $a[x]=(1,0,x_{3},1,x_{5})$.

Lemma 15**.**

Let $S\subseteq\{0,1,z\}^{n}$ be an $(n,2d+1)$-universal disjoint set. If $f\in\mathrm{DT}_{d}$ depends on $x_{i}$, then there is an assignment $a\in S$ such that $f(a[x])\in\{x_{i},\bar{x}_{i}\}$.

Proof.

If $x_{i}$ is relevant in $f$, then $f(x|_{x_{i}\leftarrow 0})\not=f(x|_{x_{i}\leftarrow 1})$. Therefore, $f^{\prime}(x)=f(x|_{x_{i}\leftarrow 0})+f(x|_{x_{i}\leftarrow 1})\not=0$. Since $f(x|_{x_{i}\leftarrow 0}),f(x|_{x_{i}\leftarrow 1})\in\mathrm{DT}_{d}$, by Lemma 2 we have $f^{\prime}(x)\in\mathrm{DT}_{2d}$. Let $x_{i_{1}}\stackrel{{\scriptstyle\xi_{1}}}{{\to}}x_{i_{2}}\stackrel{{\scriptstyle\xi_{2}}}{{\to}}\cdots{\to}x_{i_{d^{\prime}}}\stackrel{{\scriptstyle\xi_{d^{\prime}}}}{{\to}}1$, $d^{\prime}\leq 2d$ be a path in the decision tree representation of $f^{\prime}$. Such path exists since $f^{\prime}\in\mathrm{DT}_{2d}$ and $f^{\prime}\not=0$. Obviously, $i\not\in\{i_{1},\ldots,i_{d^{\prime}}\}$ since $x_{i}$ is not relevant in $f^{\prime}$. By the definition of $S$, there is an assignment $a\in S$ such that $a_{i_{1}}=\xi_{1},\ldots,a_{i_{d^{\prime}}}=\xi_{d^{\prime}}$ and $a_{i}=z$. Hence, $f^{\prime}(a)=1$ and $f^{\prime}(a[x])=1$. Therefore, either $f(a[x]|_{x_{i}\leftarrow 0})=1$ and $f(a[x]|_{x_{i}\leftarrow 1})=0$, or $f(a[x]|_{x_{i}\leftarrow 0})=0$ and $f(a[x]|_{x_{i}\leftarrow 1})=1$. As a result, we get $f(a[x])=x_{i}f(a[x]|_{x_{i}\leftarrow 1})+\bar{x}_{i}f(a[x]|_{x_{i}\leftarrow 0})\in\{x_{i},\bar{x}_{i}\}$. ∎

Figure 2 describes the algorithm. In step 1, the algorithm constructs an $(n,2d+1)$-universal disjoint set $S$. In step 2, it defines an empty set $R$ that eventually, will contain all the relevant variables. Then, in step 3, for every $a\in S$, it defines a new assignment $b$ that is derived from $a$ by assigning to zero all $a_{i}=z$ in $a$ that indicate a known relevant variable $x_{i}\in R$. Thereafter, if $f(b[1^{n}])\not=f(b[0^{n}])$, the algorithm runs Binary-Learn-Variable$(f,b[1^{n}],b[0^{n}])$. The procedure Binary-Learn-Variable satisfies the following:

Lemma 16**.**

*Let $a\in\{0,1,z\}^{n}$. If $f(a[1^{n}])\not=f(a[0^{n}])$ then, the procedure call Binary-Learn-Variable $(f,a[1^{n}],a[0^{n}])$ finds, in at most $\lceil\log n\rceil$ queries, a relevant variable $x_{j}$ that satisfies:

1. $a[1^{n}]_{j}+a[0^{n}]_{j}=1$, i.e., $a_{j}=z$.

2. $x_{j}$ is a relevant variable in $f(a[x])$ and therefore in $f$.*

The algorithm adds, in step 3.2, the learned relevant variable to the set $R$. We prove:

Lemma 17**.**

The number of times that Binary-Learn-Variable is executed is at most $V(f)$, where $V(f)$ the is the number of the relevant variables of $f$.

Proof.

Notice that after step 3.1, for every $x_{i}\in R$ we have $a_{i}=0$ and hence $b_{i}=0$ too. Therefore, for every $x_{i}\in R$, we have $b[1^{n}]_{i}=b[0^{n}]_{i}$. By Lemma 16, the relevant variable that Binary-Learn-Variable finds is not in $R$. Also, by Lemma 16, all the variables in $R$ are relevant in $f$. Thus, Binary-Learn-Variable cannot be executed more than the number of relevant variables of $f$ i.e. $V(f)$. ∎

We now show:

Lemma 18**.**

The procedure Find-Relevant returns all the relevant variables in $f$.

Proof.

Let $x_{j}$ be a relevant variable in $f$. By Lemma 15, there is an assignment $a\in S$ such that $f(a[x])\in\{\bar{x_{j}},x_{j}\}$. If $x_{j}\in R$ when the algorithm reaches $a$, then we are done. Otherwise, if $x_{j}\notin R$, then, the algorithm defines a new assignment $b$ from $a$ such that for each $x_{i}\in R$ and $a_{i}=z$ we have $b_{i}=0$, otherwise, $b_{i}=a_{i}$. In particular, $b_{j}=z$. Therefore, $f(b[x])\in\{\bar{x_{j}},x_{j}\}$. This is true because, if $c\in\{0,1,z\}^{n}$ and $f(c[x])=x_{j}$ (or $\bar{x}_{j}$), then for any $k\not=j$ where $c_{k}=z$, $f(c|_{x_{k}\leftarrow 0}[x])=f(c[x]|_{x_{k}\leftarrow 0})=x_{j}$ (or $\bar{x}_{j}$). Therefore, $f(b[1^{n}])\not=f(b[0^{n}])$ and the algorithm runs Binary-Learn-Variable$(f,b[1^{n}],b[0^{n}])$. By Lemma 16, the procedure Binary-Learn-Variable finds a relevant variable in $f(b[x])\in\{\bar{x_{j}},x_{j}\}$ and therefore finds $x_{j}$. ∎

Lemma 19**.**

The query complexity of Find-Relevant is $2^{2d+O(\log^{2}d)}\log n$.

Proof.

Steps 3.1-3.2 are executed $|S|$ times and therefore, by Lemma 7, the number of queries asked in the If condition is $2|S|=2^{2d+O(\log^{2}d)}\log n$. Moreover, by Lemma 17, Binary-Learn-Variable is executed $V(f)$ times and, by Lemma 16, in each execution it asks $\lceil\log n\rceil$ queries. Therefore, the number of queries asked in Binary-Learn-Variable is $V(f)\lceil\log n\rceil=O(2^{d}\log n)$. ∎

4.2 Learning MP

In this section we give an algorithm that learns $\mathrm{MP}_{d,s}$ with $2^{2.66d}s^{2}\log^{3}N$ queries, where $N$ is the number of variables. We first show how to learn one monomial.

The algorithm is in Figure 3. In the algorithm, the procedure Zero-Test answers “True” if $f$ is identically zero and “False” otherwise. It uses Lemma 3. Recall that, $1^{N}$ is the all one assignment of dimension $N$, $wt(b)$ is the Hamming weight of $b$ and $a*x=(a_{1}x_{1},\ldots,a_{N}x_{N})$. For two assignments $b\in\{0,1\}^{N}$ and $a\in\{0,1\}^{wt(b)}$, the assignment $a\Delta b$ is defined as follows. Let $1\leq i_{1}<i_{2}<\cdots<i_{wt(b)}$ be all the indices $j$ such that $b_{j}=1$. Then $(a\Delta b)_{i_{\ell}}=a_{\ell}$ for all $\ell=1,2,\ldots,wt(b)$ and $(a\Delta b)_{j}=0$ for all $j\not\in\{i_{1},\ldots,i_{wt(b)}\}$. For example, if $b=(10011101)$ and $a=(10110)$, then $a\Delta b=(a_{1}00a_{2}a_{3}a_{4}0a_{5})=(10001100)$.

Lemma 20**.**

If $f(b*x)\not=0$ then,

If $f((a\Delta b)*x)\not=0$ for some $a\in D$, then $wt(a\Delta b)\leq wt(b)(1-1/(2d))$. 2. 2.

If $f((a\Delta b)*x)=0$ for all $a\in D$, then $M=\wedge_{b_{i}=1}x_{i}$ is a monomial in $f$.

Proof.

Since $D$ is $(wt(b),\min(wt(b)-1,d))$-sparse all one set and $a\in D$, $wt(a\Delta b)=wt(a)\leq wt(b)(1-1/(2d))$. This implies 1. To prove 2., suppose $f((a\Delta b)*x)=0$ for all $a\in D$. Let $M=x_{i_{1}}\cdots x_{i_{d^{\prime}}}$, $d^{\prime}\leq d$ be any minimal size monomial in $f(b*x)$. In particular, $b_{i_{1}}=\cdots=b_{i_{d^{\prime}}}=1$. From the definition of $\Delta$, we have $(a\Delta b)_{i_{\ell}}=a_{j_{\ell}}$, $\ell=1,\ldots,d^{\prime}$, for some $1\leq j_{1}<j_{2}<\cdots<j_{d^{\prime}}\leq wt(b)$. We now prove that $wt(b)=d^{\prime}$. Suppose, for the sake of contradiction, that $wt(b)\geq d^{\prime}+1$. Then, $d^{\prime\prime}=\min(wt(b)-1,d)\geq d^{\prime}$. Since $D$ is $(wt(b),d^{\prime\prime})$-sparse all one set, there is an assignment $a\in D$ such that $a_{j_{\ell}}=1$ for all $\ell=1,2,\ldots,d^{\prime}$. Thus, $(a\Delta b)_{i_{\ell}}=1$ for all $\ell=1,2,\ldots,d^{\prime}$. As a result, $f((a\Delta b)*x)$ will contain the monomial $M$ and therefore is not zero. This is a contradiction. Therefore, $wt(b)=d^{\prime}$. Since $wt(b)=d^{\prime}$ and $b_{i_{1}}=\cdots=b_{i_{d^{\prime}}}=1$, we have $\wedge_{b_{i}=1}x_{i}=x_{i_{1}}\cdots x_{i_{d^{\prime}}}=M$. ∎

Lemma 21**.**

The number of iterations in step 3 is at most $O(d\log N)$.

Proof.

At the beginning $b=1^{N}$ and, by Lemma 20, at each iteration the weight of $b$ is reduced by a factor of $1-1/(2d)\leq e^{-1/(2d)}$. This implies the result. ∎

Lemma 21 implies that:

Lemma 22**.**

Algorithm Learn-Monomial deterministically learns one monomial in $f$ with $\tilde{O}(2^{2.66d})s\log^{3}N$ queries.

Proof.

By Lemma 3, each zero test takes $2^{2.66d}s\log^{2}N$ queries. By Lemma 21, the number of iterations in step 3 is at most $O(d\log N)$. ∎

We now prove:

Theorem 23**.**

There is a deterministic learning algorithm that learns the class $\mathrm{MP}_{s,d}$ in $\tilde{O}(2^{2.66d})s^{2}\log^{3}N$ queries.

Proof.

Assume we have found $t$ monomials $M_{1},\ldots,M_{t}$ of $f$ we can learn another new monomial by learning a monomial of $f^{\prime}=f+M_{1}+M_{2}+\cdots+M_{t}$. The function $f^{\prime}$ is just the function $f$ without the monomials in $M_{1},\ldots,M_{t}$. This is because $M+M=0$ in the binary field. Using Lemma 22, the result follows. ∎

Appendix A KM-algorithm

Along the discussion, we will frequently need to estimate the expected value of random variables using Chernoff bound. Chernoff bound allows us to compute the sample size needed to ensure that the estimation is close to its mean with high probability. To make the analysis easy to follow for the reader, we bring Chernoff bound in the following Lemma.

Lemma 24**.**

(Chernoff). Let $X_{1},\cdots,X_{m}$ be independent random variables all with mean $\mu$ such that for all $i$, $X_{i}\in[-1,1]$. Then, for any $\lambda>0$,

[TABLE]

A.1 The Discrete Fouriere Transform - DFT

One of the powerful techniques used in the litrature for learning $\mathrm{DT}_{d}$ is the Discrete Fourier Transform - $DFT$[18]. To simplify the analysis, it is more convenient to change the range of the Boolean functions from $\{0,1\}$ to $\{-1,+1\}$, i.e. $g:\{0,1\}^{n}\rightarrow\{-1,+1\}$ by redefining the function $g$ from the standard representation of $f$ according to $g=1-2f$. In this case, the zero-labeled leaves of $T(f)$ will be relabeled by $-1$ while the others remain untouched. Moreover, Definition 1 still holds for this case too with a minor change that $f\equiv-1$ is a decision tree instead of $f\equiv 0$. Similarily, the “$+$” sign still indicates arethmetic summation. Therefore, we can relate to $f$ as a real valued function $f:\{0,1\}^{n}\rightarrow\operatorname*{\mathbb{R}}$. The following is an example of a decision tree of depth $3$

[TABLE]

Denote by $\operatorname*{\mathbb{E}}[f]$ the expected value of $f(x)$ with respect to the uniform distribution on $x$. Along the discussion, in case the expectation is taken over uniform distribution, we will omit the distibution notation.

Let $\mathcal{F}$ denote the set of real functions on $\{0,1\}^{n}$. $\mathcal{F}$ is a $2^{n}$-dimensional real vector space with inner product defined as:

[TABLE]

The norm of a function $f$ is defined by

[TABLE]

For each $a\in\{0,1\}^{n}$ define the parity function $\chi_{a}(x)$:

[TABLE]

The functions $\{\chi_{a}\}_{{a}\in\{0,1\}^{n}}$ are an orthonormal basis for $\mathcal{F}$. Hence, for all $f\in\mathcal{F}$ we can write

[TABLE]

for $\hat{f}(a)\in\operatorname*{\mathbb{R}}$. The coefficients $\hat{f}(a)$ are called Fourier coeffcients and,

[TABLE]

By Parsaval’s Theorem, we get that $\operatorname*{\mathbb{E}}[f^{2}]=\sum_{a\in\{0,1\}^{n}}\hat{f}^{2}(a)$. For Boolean functions defined over the range $\{-1,+1\}$ we get that,

[TABLE]

We now show that:

Lemma 25**.**

Let $f=\sum_{a\in A}\lambda_{a}\chi_{a}(x)$ be any function where $\lambda_{a}\not=0$ for every $a\in A$. Then, the relevant variables of $f$ can be found in time $O(|A|n)$. Moreover, for each relevant variable $x_{i}$, an assignment $b$ such that $f(w|_{x_{i}\leftarrow 0})\not=f(w|_{x_{i}\leftarrow 1})$, i.e., a witness for $x_{i}$, can be found in time $O(|A|n)$.

Proof.

First, since $\chi_{a}$ are orthonormal basis, $f=0$ if and only if $A=\varnothing$. Second, to find an assignment $w$ such that $f(w)\not=0$, we find $\xi_{1}\in\{0,1\}$ such that $f(x|_{x_{1}\leftarrow\xi_{1}})\not=0$ and then recursively do that for $x_{2},\ldots,x_{n}$. This takes $O(|A|n)$ steps.

Since $f=f_{1}+(-1)^{x_{i}}f_{2}$ where $f_{1}=\sum_{a\in A,a_{i}=0}\lambda_{a}\chi_{a}(x)$ and $f_{2}=\sum_{a\in A,a_{i}=1}$ $\lambda_{a}\chi_{a}(x|_{x_{i}\leftarrow 0})$, we have $f(x|_{x_{i}\leftarrow 0})=f(x|_{x_{i}\leftarrow 1})$ if and only if $f_{2}=0$. That is, $\{a\in A|a_{i}=1\}=\varnothing$. Therefore, $x_{i}$ is relevant in $f$ if and only if $a_{i}=1$ for some $a\in A$. Thus, the set of relevant variables of $f$ is $X=\cup_{a\in A}\cup_{a_{i}=1}\{x_{i}\}$.

To find a witness for $x_{i}$ we need to find an assignment $b$ such that $f(b|_{x_{i}\leftarrow 1})\not=f(b|_{x_{i}\leftarrow 1})$. That is $f(b|_{x_{i}\leftarrow 1})-f(b|_{x_{i}\leftarrow 1})\not=0$. Consider the function $g(x)=f(x|_{x_{i}\leftarrow 1})-f(x|_{x_{i}\leftarrow 1})$. Then we need to find an assignment $b$ such that $g(b)\not=0$. This can be done as above in $O(|A|n)$ steps. ∎

A.2 From Decision Tree to DFT Representation

There is a standard way to transform a Boolean function represented as a decision tree to DFT fomulation. Definition 1 implies that every decision tree of size $s$ and depth $d$ can be written as a sum (in $\operatorname*{\mathbb{R}}$) of $s$ terms each of size at most $d$. Formally,

[TABLE]

such that for all $1\leq i\leq s$,

[TABLE]

for $d^{\prime}\leq d$, and $\ell_{i_{j}}=x_{i_{j}}$ or $\ell_{i_{j}}=\overline{x}_{i_{j}}$. Moreover, the parameter $\alpha_{i}$ is the value of the leaf in the path defined by the term $T_{i}$ in $T(f)$.

To formulate $f$ in the DFT representation, it is enough to rewrite each term $T_{i}$ and then sum the whole transformed terms into one formula. Over the real numbers $\operatorname*{\mathbb{R}}$, any positive literal $x_{i_{j}}$ in $T_{i}$ can be expressed as $(1-(-1)^{x_{i_{j}}})/2$. In a similar fashion, any negative literal $\overline{x}_{i_{j}}$ translates to $(1+(-1)^{x_{i_{j}}})/2$. We apply this translation to all the terms $T_{i}$. The result is a DFT representation of $f$. We demonstrate the process with an example. Let $f\in\mathrm{DT}_{3}$ be the decision tree described in (2). The function $f$ can be written as,

[TABLE]

where the first three terms comply with the positive pathes in the decision tree and the other two comply with the negative ones. We start with $x_{2}x_{3}$:

[TABLE]

Similarily, we can rewrite $\overline{x}_{2}x_{1}$,

[TABLE]

To get the full DFT expression of $f$, we continue in the same fashion for the other three terms. The construction described above imply the following results:

Lemma 26**.**

Let $T$ be a term of size $d^{\prime}$ then,

$T$* has a DFT representation that contains $2^{d^{\prime}}$ non-zero coefficients $\hat{f}(a)$.* 2. 2.

Each coefficient $\hat{f}(a)$ has the value $\pm\frac{1}{2^{d^{\prime}}}$. 3. 3.

Each nonzero coefficient $\hat{f}(a)$ of $T$ satisfies $wt(a)\leq d^{\prime}$.

Proof.

The proof follows directly from the construction of DFT from $T$. ∎

Lemma 27**.**

Let $f\in DT_{d}$, then for all ${a}\in\{0,1\}^{n}$,

[TABLE]

for some $-2^{d}\leq k\leq 2^{d}.$

Proof.

First notice that by Lemma 26, the Fourier coefficients of each term of size at most $d$ is of the form $k/2^{d}$. Therefore, the Fourier coefficients of the sum of terms of size at most $d$ is of the same form. The fact that $-2^{d}\leq k\leq 2^{d}$ follows from the fact that $|\hat{f}(a)|\leq 1$ which follows from Parsaval (3). ∎

Using the fact that any $f\in\mathrm{DT}_{d}$ has at most $2^{d}$ terms each of size at most $d$ we can conclude:

Corollary 28**.**

Let $f\in\mathrm{DT}_{d}$, then $f$ has a DFT representation that contains at most $2^{2d}$ non-zero coefficients $\hat{f}(a)$ each having a value in $\{\pm k/2^{d}|k\in[2^{d}]\}$ and Hamming weight $wt(a)\leq d$.

A.3 The KM-algorithm

Kushilevitz and Mansour [20] and Jackson  [21] gave an adaptive algorithm that finds the non-zero coefficients in $poly(2^{d},n)$ time and membership queries. Kushilevitz and Mansour algorithm (KM-algorithm) is a recursive algorithm that is given as an input an access to a membership query oracle $MQ_{f}$, confidence $\delta$ and a string $\alpha\in\{0,1\}^{r}$. Initially, KM-algorithm runs on the empty string $\alpha=\varepsilon$. The fundamental idea of the algorithm is based on the fact that for any $\alpha\in\{0,1\}^{r}$ it can be shown that [21],

[TABLE]

where for $y=(y_{1},\cdots,y_{s})$ and $x=(x_{1},\cdots,x_{t})$, $yx=(y_{1},\cdots,y_{s},x_{1},\cdots,x_{t})$. For Boolean functions $f:\{0,1\}^{n}\rightarrow\{-1,+1\}$, (3) and (4) imply that $F_{\alpha}\leq 1$ for all $\alpha$ where equality holds for $\alpha=\varepsilon$. Furthermore, for $f\in\mathrm{DT}_{d}$ we prove,

Lemma 29**.**

Let $f\in\mathrm{DT}_{d}$. Then, for all $\alpha\in\{0,1\}^{r}$, $r\leq n$ there is $\ell\in[2^{2d}]\cup\{0\}$ such that,

[TABLE]

for $\ell\in[2^{2d}]\cup\{0\}.$

Proof.

Follows from Corollary 28 and Parsaval (3). ∎

KM-algorithm uses divide and conquer technique with the identity in (4) to find the non-zero coefficients. Firstly, the algorithm computes $F_{0}$ and $F_{1}$ and initiates $T_{1}=\{\xi\in\{0,1\}|F_{\xi}\neq 0\}$. At some stage, the algorithm holds a set $T_{r}=\{\alpha\in\{0,1\}^{r}|F_{\alpha}\neq 0\}$ . For each $\alpha\in T_{r}$, the algorithm computes $F_{\alpha 0}$ and $F_{\alpha 1}$. Since $F_{\alpha}=F_{\alpha 0}+F_{\alpha 1}\neq 0$, at least one of the terms is not zero. Therefore, the algorithm defines $T_{r+1}=\{\alpha\xi|\alpha\in T_{r},\xi\in\{0,1\},F_{\alpha\xi}\neq 0\}\neq\O$. Using (4), when $r=n$ we can conclude that $F_{\alpha}=\hat{f}(\alpha)^{2}.$ That is, on the algorithm’s termination, $T_{n}=\{\alpha\in\{0,1\}^{n}|\hat{f}(\alpha)^{2}\neq 0\}$ includes all the non-zero Fourier coefficients of $f$. Observe that for $f\in\mathrm{DT}_{d}$, Corollary 28 implies that $f$ has at most $2^{2d}$ non-zero coefficients. Hence, the number of elements in $T_{r}$ is at most $2^{2d}$ for all $r$. The following lemma summarizes this observation.

Lemma 30**.**

[20]**. Let $f\in\mathrm{DT}_{d}$. Then, for any $1\leq r\leq n$, $F_{\alpha}\neq 0$ for at most $2^{2d}$ strings $\alpha\in\{0,1\}^{r}$.

The immediate result of Lemma 30 is that KM-algorithm computes $F_{\alpha}$ at most $n\cdot 2^{2d}$ times. Furthermore, from Chernoff bound it follows that:

Lemma 31**.**

Let $f\in\mathrm{DT}_{d}$. Let $\alpha\in\{0,1\}^{r}$ and $1\leq r\leq n$. There is an algorithm $\cal{A}$ that with probability at least $1-\delta$ exactly computes the value of $F_{\alpha}$. Moreover, the algorithm asks $O\left(2^{4d}\cdot\log(\frac{1}{\delta})\right)$ membership queries and runs in time $O\left(2^{4d}\cdot n\cdot\log(\frac{1}{\delta})\right)$.

Proof.

Let $\alpha\in\{0,1\}^{r}$. Chernoff bound can be used to calculate the expectation from (4). Let $W_{1},\cdots,W_{m}$ be $m$ random variables such that $W_{i}=f(y^{(i)}x^{(i)})f(z^{(i)}x^{(i)})\chi_{\alpha}(y^{(i)})\chi_{\alpha}(z^{(i)})\in\{-1,1\}$ for random uniform $y^{(i)},z^{(i)}\in\{0,1\}^{r}$ and $x^{(i)}\in\{0,1\}^{n-r}$. Using Chernoff bound with $\lambda=\frac{1}{2^{2d+1}}$ in (1) gives,

[TABLE]

By choosing $m=O(2^{4d}\cdot\log({1}/{\delta}))$ it follows that with probability at least $1-\delta$,

[TABLE]

Furthermore, Lemma 29 states that the possible values of $F_{\alpha}$ vary in steps of $1/2^{2d}$. Since the error is smaller than $\frac{1}{2}\cdot\frac{1}{2^{2d}}$, the exact value of $F_{\alpha}$ can be found by rounding to the nearest number of the form $\ell/2^{2d}$ where $\ell\in[2^{2d}]\cup\{0\}$.

Each membership query requires generating $O(n)$ random bits, therefore the time complexity is $O(2^{4d}\cdot n\cdot\log({1}/{\delta}))$. ∎

From Lemma 31, it can be concluded that,

Theorem 32**.**

[20]**. There is an adaptive Monte Carlo learning algorithm $\cal{B}$, that with probability at least $1-\delta$ exactly learns $\mathrm{DT}_{d}$. Moreover, the algorithm runs in time $O(2^{6d}\cdot n^{2}\cdot(\log({n}/{\delta})+d))$ and asks $O(2^{6d}\cdot n\cdot(\log({n}/{\delta})+d))$ membership queries.

Proof.

As mentioned before, the algorithm computes $F_{\alpha}$ at most $n\cdot 2^{2d}$ times. By the union bound, to get a failure probability of at most $\delta$, we need to evaluate each $F_{\alpha}$ with a failure probability ${\delta}/({n\cdot 2^{2d}})$ using algorithm $\cal{A}$ from Lemma 31. This implies that each evaluation of $F_{\alpha}$ requires $O(2^{4d}\cdot(\log({n}/{\delta})+d))$ membership queries and runs in time $O(2^{4d}\cdot n\cdot(\log({n}/{\delta})+d))$ . Since algorithm $\cal{B}$ invokes $\cal{A}$ at most $n\cdot 2^{2d}$ times, the theorem follows.

∎

A.4 Derandomization

In this subsection, we describe KM-algorithm derandomization in details. We start by giving some preliminaries needed for the analysis.

A.4.1 Preliminaries

Kushilevitz and Mansour use the method of $\lambda$-bias distributions to derandomize their algorithm. The following definitions are useful to review this derandomization method.

For any set $R$ we will use the notation $\mathbf{\operatorname*{\mathbb{E}}}_{R}[\cdot]$ to denote $\mathbf{\operatorname*{\mathbb{E}}}_{x\in R}[\cdot]$ with the uniform distribution over the elements of $R$. Let $S=\{0,1\}^{n}$ and let $S^{\prime}\subseteq S$,

Definition 2**.**

We say that $S^{\prime}$ is a $d$-wise independent set if for all uniformly random $x\in S^{\prime}$ and all $1\leq i_{1}<\cdots<i_{d}\leq n$ and all $\xi_{1},\cdots,\xi_{d}$ we have

[TABLE]

Alon et al.[28] showed:

Lemma 33**.**

Let $w=\lceil\log n\rceil$, $2s+1\geq d$ and $m=ws+1$. Then, there is an $m\times n$ matrix $M$ over the field $F_{2}$ that can be constructed in $poly(n)$ time such that $S^{\prime}=\{xM|x\in\{0,1\}^{m}\}$ is $d$-wise independent set.

Definition 3**.**

We say that $S^{\prime}$ is $\lambda$-biased with respect to linear tests if for all $\alpha\in\{0,1\}^{n}\backslash\{0^{n}\}$ it holds that,

[TABLE]

Moreover, Ta-Shma[22] proves that:

Lemma 34**.**

There is a $\lambda$-biased with respect to linear tests $S^{\prime}$ of size

[TABLE]

that can be constructed in polynomial time $poly(n,1/\lambda)$.

Definition 4**.**

We say that $S^{\prime}$ is $\lambda$-biased with respect to linear tests of size at most $d$ if for all $\alpha\in\{0,1\}^{n}\backslash\{0^{n}\}$ such that $wt(\alpha)\leq d$ it holds that,

[TABLE]

Naor and Nir[29] prove that:

Lemma 35**.**

Given a matrix $M$ as in Lemma 33 of size $m\times n$ and a $\lambda$-biased with respect to linear tests set $\hat{S}\subseteq\{0,1\}^{m}$, one can, in polynomial time, construct a $\lambda$-biased set with respect to linear tests of size at most $d$ of size $|\hat{S}|$.

Taking $s=\lceil(d-1)/2\rceil$ in Lemma 33, we get $m=O(d\log n)$ and by Lemma 34 and 35 we conclude that:

Lemma 36**.**

There is a $\lambda$-biased with respect to linear tests of size at most $d$, $S^{\prime}$ of size

[TABLE]

that can be constructed in polynomial time $poly(n,1/\lambda)$.

A.4.2 KM-algorithm derandomization via $\lambda$-bias sample spaces

The appropriateness of algorithm derandomization via $\lambda$-bias spaces requires showing that the output of the algorithm when its coin tosses are chosen from a uniform distribution, and the output of the algorithm when its coin tosses are chosen from $\lambda$-bias distribution are very similar. If this holds, a deterministic version of KM algorithm will replace the Chernoff-based computation of $F_{\alpha}$ by taking expectation over all elements of the $\lambda$-bias sample space. Consequently, a polynomially-small bias sample space is a mandatory requirement. To show “similarity” between the randomized algorithm and the deterministic one, we start with the following lemmas. By Parsaval’s Theorem we have:

Lemma 37**.**

Let $f:S\rightarrow\mathbb{R}$ and $S=\{0,1\}^{n}$. Then,

[TABLE]

Definition 5**.**

Let $f$ be any Boolean function. $L_{1}$-norm of $f$ is defined as,

[TABLE]

By subadditivity of the absolute value we have $L_{1}(f+g)\leq L_{1}(f)+L_{1}(g)$ and by Lemma 26, for any term $T$, $L_{1}(T)=1$. Since each $f\in\mathrm{DT}_{d}$ is a sum of at most $2^{d}$ terms we have:

Lemma 38**.**

Let $f\in\mathrm{DT}_{d}$. Then, $L_{1}(f)\leq 2^{d}$.

Lemma 39**.**

Let $f:S\rightarrow\mathbb{R}$ for $S=\{0,1\}^{n}$, and $S^{\prime}\subseteq S$. Let

[TABLE]

then,

[TABLE]

Proof.

Using Lemma 37 we have,

[TABLE]

Thus we can say,

[TABLE]

∎

Definition 6**.**

Let $S_{1}=\{0,1\}^{k}$ and $S_{2}=\{0,1\}^{n-k}$ for $0\leq k\leq n$. Let $S_{1}^{\prime}\subseteq S_{1}$, $\alpha,y\in S_{1}$ and $x\in S_{2}$. Define $h_{S_{1}^{\prime}}:S_{2}\rightarrow\mathbb{R}$ such that,

[TABLE]

When choosing $S_{1}^{\prime}=S_{1}$, we will use the notation $h=h_{S_{1}}$. In this case $h$ is defined by taking the expectation on uniform distribution over $S_{1}$.

Lemma 40**.**

Let $S=\{0,1\}^{n}$ and $S_{1}$, $S_{2}$ as in Definition 6. Let $S_{1}^{\prime}\subseteq S_{1}$. Let $h_{S_{1}^{\prime}}$ and $h=h_{S_{1}}$ be as defined in (6). Then,

$L_{1}(h)\leq L_{1}(f).$** 2. 2.

[TABLE]

Proof.

We first prove 1. Using Definition 6 and the Fourier transform of $f$ and the fact that $\mathbf{\operatorname*{\mathbb{E}}}_{S_{1}}[\chi_{a}(y)]=0$ for all $a\neq 0$ when the expectation is taken over uniform distribution we get,

[TABLE]

The last equation follows because $\operatorname*{\mathbb{E}}[\chi_{a_{1}+\alpha}(y)]=0$ for $a_{1}\not=\alpha$. Hence, we get that $\hat{h}(a)=\hat{f}(\alpha a)$. Therefore,

[TABLE]

We now prove 2. Similarily, we can say,

[TABLE]

∎

Lemma 41**.**

Let $S_{1}^{\prime}\subseteq S_{1}$, and $S_{2}^{\prime}\subseteq S_{2}$ and $h_{S_{1}^{\prime}},h_{S_{1}}=h$ be as in Definition 6. Let $\lambda_{1}=\mathop{max}_{c\in S_{1},c\neq 0}|\mathop{\mathbf{\operatorname*{\mathbb{E}}}}_{S_{1}^{\prime}}[\chi_{c}]|$ and $\lambda_{2}=\mathop{max}_{c\in S_{2},c\neq 0}|\mathop{\mathbf{\operatorname*{\mathbb{E}}}}_{S_{2}^{\prime}}[\chi_{c}]|$ then,

[TABLE]

Proof.

By the subadditivity of the absolute value function,

[TABLE]

Since $-1\leq h_{S_{1}^{\prime}}\leq 1$ and $-1\leq h\leq 1$ we can see that,

[TABLE]

By Lemma 40,

[TABLE]

Hence, using (8) and (A.4.2) we conclude,

[TABLE]

On the other hand, by lemmas 39 and 40 we can say that,

[TABLE]

By combining (7), (10) and (11) the Lemma follows. ∎

Using Definition 6 we can rewrite Equation 4:

[TABLE]

Derandomization via bias spaces technique is done by choosing polynomially small $S_{1}^{\prime}\subseteq S_{1}$ and $S_{2}^{\prime}\subseteq S_{2}$ such that the expectations needed to evaluate $F_{\alpha}$ can be done by taking the expectation over the small domains $S_{1}^{\prime}$ and $S_{2}^{\prime}$. Let $F_{\alpha}^{\prime}$ denote the estimatioin of $F_{\alpha}$ derived from (12) by taking the expectations over $S_{1}^{\prime}$ and $S_{2}^{\prime}$. That is,

[TABLE]

Let $\Delta(F_{\alpha})$ denote the error in evaluating $F_{\alpha}$. Namely,

[TABLE]

To get good approximation for $F_{\alpha}$, we require the value $\Delta(F_{\alpha})$ to be small with high probability. Moreover, Lemma 29 implies that requiring $\Delta(F_{\alpha})=\frac{1}{2^{2d+1}}$ and rounding the value of $F_{\alpha}^{\prime}$ to the closest $\ell/2^{2d}$ gives accurate evaluation of $F_{\alpha}$. By lemmas 38 and 41 we choose $S_{1}^{\prime}$ and $S_{2}^{\prime}$ such that,

[TABLE]

for $\lambda_{1}=\mathop{max}_{c\in S_{1},c\neq 0}|\mathop{\mathbf{\operatorname*{\mathbb{E}}}}_{S_{1}^{\prime}}[\chi_{c}(x)]|$ and $\lambda_{2}=\mathop{max}_{c\in S_{2},c\neq 0}|\mathop{\mathbf{\operatorname*{\mathbb{E}}}}_{S_{2}^{\prime}}[\chi_{c}(x)]|$. The number of membership queries needed in these settings is $|S_{1}^{\prime}|\cdot|S_{2}^{\prime}|$ for each estimation of $F_{\alpha}$. It is easy see that we get optimal sizes when $\lambda_{1}=1/2^{3d+3}$ and $\lambda_{2}=1/2^{4d+2}$.

Corollary 28 implies that the non-zero Fourier coefficients of $f\in\mathrm{DT}_{d}$ have Hamming weight of at most $d$. All the assignments $c$ in the expectations of $\chi_{c}$ are of the form $c=a+b$ where $\hat{f}(a),\hat{f}(b)\not=0$ and therefore $wt(c)\leq 2d$. See (5) and (A.4.2). Therefore, for $\lambda_{1}$ we need a set that is $\lambda_{1}$-biased with respect to linear tests of size at most $2d$, and for $\lambda_{2}$, a set that is $\lambda_{2}$-biased with respect to linear tests of size at most $2d$. By Lemma 36, such sets can be constructed in $poly(2^{d},n)$ time, and are of sizes $d{2^{6d+o(d)}\log n}{}$ and $d{2^{8d+o(d)}\log n}{}$ respectively. This gives $\tilde{O}(2^{14d+o(d)})\log^{2}n$ queries to exactly find each $F_{\alpha}$.

As mentioned in the KM-algorithm we need to compute $F_{\alpha}$ for $n2^{2d}$ $\alpha$’s and therefore

Theorem 42**.**

There is an adaptive deterministic learning algorithm that exact learns $\mathrm{DT}_{d}$ in $poly(2^{d},n)$ time and asks $\tilde{O}(2^{16d+o(d)})n\cdot\log^{2}n$ membership queries.

Using the reduction introduced in [23] for decision trees we have,

Theorem 43**.**

There is an adaptive deterministic learning algorithm that exact learns $\mathrm{DT}_{d}$ in $poly(2^{d},n)$ time and asks $\tilde{O}(2^{18d+o(d)})\cdot\log n$ membership queries.