Exact Learning from an Honest Teacher That Answers Membership Queries

TL;DR

This paper surveys methods for exactly learning functions from an honest teacher through membership queries, highlighting known results, techniques, and open challenges in the field.

Contribution

It provides a comprehensive overview of existing literature, techniques, and open problems in exact learning from membership queries.

Findings

Summarizes key results in exact learning from membership queries.

Discusses various techniques used in the literature.

Identifies open problems and future research directions.

Abstract

Given a teacher that holds a function $f:X\to R$ from some class of functions $C$. The teacher can receive from the learner an element~$d$ in the domain $X$ (a query) and returns the value of the function in $d$, $f(d)\in R$. The learner goal is to find $f$ with a minimum number of queries, optimal time complexity, and optimal resources. In this survey, we present some of the results known from the literature, different techniques used, some new problems, and open problems.

Equations

$$C^D=\left\{\overline{f({\bar{x}}_1,\dots ,{\bar{x}}_n)}|f\in C\right\}.$$

$$f^{\prime }=$$

$$f=\underset{i=1}{\overset{s}{\bigvee }}{T}_i$$

$$f=M_1+M_2+\cdots +M_s$$

$$f(x_1,\dots ,x_n)=A_1(x_1)A_2(x_2)\cdots A_n(x_n)$$

$$f(x_{1},\ldots,x_{n})=\left\{\begin{array}[]{ll}1&\mbox{if}\ w_{1}x_{1}+w_{2}x_{2}+\cdots+w_{n}x_{n}\geq u\\ 0&\mbox{otherwise}\end{array}\right.$$

$$f=\sum _{i\in I}{a}_i{x}_1^{i_1}\cdots x_n^{i_n}$$

$$f(x_1,\dots ,x_n)=A_1(x_1)A_2(x_2)\cdots A_n(x_n)$$

$$\begin{array}[]{rcccl}&&{\rm AF}&\rightarrow&{\rm AC}\\[-6.544pt] &\nearrow&&\nearrow&\\[-7.96674pt] {\rm MP}&\rightarrow&{\rm MAF}&&\end{array}$$

$${\mathrm{OPT}}_{\mathrm{NAD}}^{\mathrm{ET}}(C)=2\cdot {\mathrm{OPT}}_{\mathrm{NAD}}(C).$$

$${\mathrm{OPT}}_{\mathrm{AD}}^{\mathrm{ET}}(C)=2\cdot {\mathrm{OPT}}_{\mathrm{AD}}(C).$$

$$\mathrm{TD}(H,C)=\underset{h\in H}{\max }{\mathrm{OPT}}^{\mathrm{HS}}(C-h)$$

$${\mathrm{OPT}}_{\mathrm{NAD}}(C)={\mathrm{OPT}}^{\mathrm{HS}}(C-C)$$

$${\mathrm{OPT}}_{\mathrm{AD}}(C)\ge \log |C|.$$

$${\mathrm{OPT}}_{\mathrm{AD},\mathrm{LV}}(C)\ge {\mathrm{OPT}}_{\mathrm{AD},\mathrm{MC}}(C)\ge \log |C|-1.$$

$$|\{f\in C|(\forall x\in T_h)h(x)=f(x)\}|\le 1.$$

$$\mathrm{ETD}(C)=\underset{h\in 2^X}{\max }T(C,h).$$

$$\mathrm{ETD}(C)\le |C|.$$

$$\mathrm{ETD}(C)\le \mathrm{TD}(2^X,C)\le \mathrm{ETD}(C)+1.$$

$${\mathrm{OPT}}_{\mathrm{AD}}(C)\le \frac{\mathrm{ETD}(C)}{\log \mathrm{ETD}(C)}\log |C|+\mathrm{ETD}(C)\le \frac{2\cdot \mathrm{ETD}(C)}{\log \mathrm{ETD}(C)}\log |C|$$

$${\mathrm{OPT}}_{\mathrm{AD}}(C)\ge \max (\mathrm{ETD}(C),\log |C|).$$

$$ϵ|C_i|\le |\{f\in C_i|f(a)=0\}|\le (1-ϵ)|C_i|.$$

$$|C_{i+1}|\le (1-ϵ)|C_i|.$$

$$|C_{i+1}|\le ϵ|C_i|.$$

$$\frac{\log |C|-k\log (1/ϵ)}{\log (1/(1-ϵ))}$$

$$(1+o(1))\frac{\mathrm{ETD}(C)}{\log \mathrm{ETD}(C)}\log |C|.\text{squareforqed}$$

$${\mathrm{OPT}}_{\mathrm{AD}}(C_{t,\ell })=\Omega \left(\frac{\mathrm{ETD}(C_{t,\ell })}{\log \mathrm{ETD}(C_{t,\ell })}\log |C_{t,\ell }|\right).$$

$${\mathrm{OPT}}_{\mathrm{AD}}(C)=\underset{x\in X}{\min }\max ({\mathrm{OPT}}_{\mathrm{AD}}(C_{x,0}),{\mathrm{OPT}}_{\mathrm{AD}}(C_{x,1}))$$

$$\min \left(1+\frac{\log |C|}{\log \mathrm{ETD}(C)},\frac{\mathrm{ETD}(C)}{\log |C|}+\frac{\mathrm{ETD}(C)}{\log \mathrm{ETD}(C)}\right)$$

Full text

In this survey, we present some of the results known from the literature, different techniques used, some new problems, and open problems.

1. 1 Introduction

2. 1.1 The Learning Model

3. 1.2 Domain and Range

4. 1.3 Classes of Functions

5. 1.4 Learning Algorithms and Complexity

6. 1.5 Polynomially, Efficiently and Optimally Learnable

7. 1.6 Strongly Polynomially, Efficiently and Optimally Learnable

8. 1.7 Testing Problems

9. 2 Bounds on OPT for Boolean Functions and Algorithms

10. 2.1 OPT for Adaptive Algorithms

11. 2.2 Constructing Adaptive Algorithms

12. 2.3 OPT for Non-Adaptive Algorithms

13. 2.4 OPT for Classes of Small VC-dimension

14. 3 Reductions

15. 3.1 Reductions for Adaptive Algorithms

16. 3.2 Reductions for Strong Adaptive Algorithms

17. 3.3 Reductions for Non-Adaptive Algorithms

18. 3.4 Reductions from the Exact Learning Model

19. 3.5 Reductions from the PAC Learning Model

20. 4 Learning $d$-MClause and Group Testing

21. 4.1 Group Testing and Applications

22. 4.2 Known Results for Learning $d$-MClause

23. 4.3 Bounds for OPT($d$-MClause)

24. 4.4 Non-Adaptive Learning $d$-MClause

25. 4.5 Adaptive Learning $d$-MClause

26. 4.6 Two-Round Learning

27. 4.7 Other Related Problems

28. 5 Learning $s$-Term $r$-Monotone DNF

29. 5.1 Learning a Hypergraph and its Applications

30. 5.2 Cover Free Families

31. 5.3 Non-Adaptive Learning $s$-Term $r$-MDNF

32. 5.4 Adaptive Learning $s$-Term $r$-MDNF

33. 5.5 Learning Subclasses of $s$-term $r$-MDNF

34. 6 Learning Decision Tree

35. 6.1 Bounds on ${\rm OPT}({\rm DT}_{d})$

36. 6.2 Adaptive Learning Decision Tree

37. 6.3 Non-Adaptive Learning Decision Tree

38. 7 Other Results

39. 7.1 Other Boolean Classes

40. 7.2 Classes of Arithmetic Functions

41. 8 Non-Honest Teacher

42. 8.1 Models of Non-Honest Teacher

43. 8.2 Some Results in Learning with Non-honest Teacher

44. 9 Problems and Open Problems

1 Introduction

Robert Dorfman’s paper in 1943 introduced the field of Group Testing. The motivation arose during the Second World War when the United States Public Health Service and the Selective service embarked upon a large scale project. The objective was to weed out all syphilitic men called up for induction. However, syphilis testing back then was expensive and testing every soldier individually would have been very cost heavy and inefficient. A basic breakdown of a test is: Draw sample from a given individual, perform required tests and determine the presence or absence of syphilis. Suppose we have $n$ soldiers. Then this method of testing leads to $n$ tests. Our goal is to achieve effective testing in a scenario where it does not make sense to test $100,000$ people to get (say) $10$ positives. The feasibility of a more effective testing scheme hinges on the following property. We can combine blood samples and test a combined sample together to check if at least one soldier has syphilis [277].

Let $S$ be the set of the $n$ soldiers and let $I\subseteq S$ be the set of the sick soldiers. Suppose we know that the number of sick soldiers, $|I|$, is bounded by some integer $d$. If $T$ is the set of soldiers for which their blood samples is combined, then the test is positive if and only if $I\cap T$ is not empty. Thus, we can regard the set of sick soldiers $I$ as a Boolean function $f_{I}:2^{S}\to\{0,1\}$ and the answer of the test “Is $I\cap T$ is not empty” as $f_{I}(T)=1$ if and only if $I\cap T\not=\O$. The goal is to identify the function $f_{I}$ (and therefore the sick soldiers) from a minimal number of substitutions (tests) and optimal time. We can also identify the set of soldiers with the set $[n]:=\{1,2,\ldots,n\}$ and regard each test as an assignment $a\in\{0,1\}^{n}$, where $a_{i}=1$ if and only if the $i$th soldier blood is in the test. Then the set $S=\{0,1\}^{n}$ is the set of all possible tests. The set of sick soldiers $I\subseteq[n]$ corresponds to a Boolean function $f^{\prime}_{I}:S\to\{0,1\}$ where $f^{\prime}_{I}(x_{1},\ldots,x_{n})=\bigvee_{i\in I}x_{i}$ and $\vee$ is the Boolean or (disjunction). So this problem is also equivalent to the problem of identifying, a hidden Boolean conjunction of up to $d$ variables, with a minimal number of substitutions and optimal time.

Another interesting problem is the problem of learning decision tree with a minimal number of queries. Let’s say one has a restaurant and she wants to learn each customer tastes preference in food. For every customer, she offers a sample of a meal that was never ordered by the customer before and then receives some feedback. The customer tastes preference depends on some attributes of the food. For example, “sweet”, “sour”, “salty”, “umami”, “bitter”, “greasy”, “hot” etc. Those are the attributes. The goal is to learn (find out) the customer tastes preference from a minimal number of samples. Each sample can be regarded as a set of attributes. The customer tastes preference is the objective function. This function depends on the attributes, and the value of the function is the customer feedback. In many cases, the target function can be described as a decision tree. See the example in Figure 1.

In the following subsection, we give a framework to the above problems and many other similar problems.

1.1 The Learning Model

Let the domain (instance space) be the set $X_{n}\in\{X_{j}\}_{j\geq 1}$ and the range be the set $R_{n}\in\{R_{j}\}_{j\geq 1}$. Let $C_{n}$ be a class of representations of functions $f:X_{n}\to R_{n}$ (target class, concept class). Given a teacher (black box, opponent player, responder) that holds a (target) function (concept) $f$ from the class $C_{n}$. The learner (player, questioner) can ask the teacher *membership queries (for Boolean functions. i.e. $R_{n}=\{0,1\}$) or substitution queries *(for non-Boolean functions), i.e., it can send the teacher an element $d$ of the domain $X_{n}$ and the teacher returns $f(d)$. The learner knows $\{C_{j},X_{j},R_{j}\}_{j\geq 1}$. Our (the learner) ultimate goal is to write an (exact) learning algorithm that learns $C=\cup_{j\geq 1}C_{j}$ with a minimum number of queries and optimal resources. That is,

Input: The learning algorithm receives the input $n$ and has access to an oracle MQf that answers membership/substitution queries for the target function $f\in C_{n}$. 2. 2.

Query complexity: It asks the teacher a minimum number of membership/substitution queries. 3. 3.

Exact learning: It either learns (finds, outputs) $g\in C_{n}$ such that $g$ is logically equivalent to $f$, $g=f$, (proper learning) or learns $h\in H_{n}\supseteq C_{n}$ such that $h=f$ (non-proper learning from $H_{n}$). 4. 4.

Resources Complexity: It runs in linear/polynomial/optimal time complexity, optimal space complexity, an optimal number of random bits or/and other optimal resources.

The following decision problems are also considered in the literature

Equivalent test: Given two teachers that have two functions from $C_{n}$ each. Test whether the two functions are equivalent. 2. 2.

Identity test from $H_{n}$: Given a teacher that has a function $f$ from $C_{n}$. Given a function $h\in H_{n}$. Test whether $f=h$. 3. 3.

Zero test: Given a teacher that has a function $f$ from $C_{n}$. Test whether $f=~{}0$.

The number of queries (query complexity) and the resources complexities are expressed as functions in $n$ and some other parameters that depend on the class being learned. In the literature, there are many other variations of the above problems, and we will mention some of them in this survey.

This problem has different names in different areas: Conditional and unconditional Tests [208], Combinatorial Search [177], Interpolation [75], Combinatorial Group Testing [113], Exact Learning from Membership Queries [2], Inferring [152], Identifying [146], Test Recognition [138], Active Learning [243], Reconstruction [195] and Guessing Game [273]. The decision problems are also called Testing, Functional Verification, Teaching, Hitting Set, and when $f$ is polynomial, it is called Black Box polynomial identity testing (PIT) [239, 255].

There are many other learning models, but, throughout this survey, when we say exact learning or learning we mean exact learning from membership queries or substitution queries only.

In this survey, we present some of the results known from the literature, different techniques used and some open problems.

1.2 Domain and Range

Throughout this survey, we will omit the subscript $n$ from $C_{n},X_{n}$ and $R_{n}$. In principle, the domain $X$ and the range $R$ can be any two sets, but since mathematical models can explain many natural phenomena, most of the sets considered in the literature are either finite or have some algebraic structure such as rings, fields, integers and real numbers.

Therefore, the domains and ranges considered in the literature are: The Boolean set that can be either $\{0,1\}$, $\{-1,+1\}$, $\{+,-\}$ or the binary field $F_{2}$. The finite discrete set can be any finite set or a finite set with some algebraic structure such as the ring $Z_{n}$ of integers modulo $n$, or the finite field $F_{q}$ with $q$ elements ($q$ is a power of prime). The infinite discrete set can be any countably infinite set such as the set of integers $Z$ or the set of rational numbers $Q$. The infinite set (uncountable) can be any set with some algebraic structure such as the real numbers $\Re$ or the complex numbers ${\cal C}$. Also, the cartesian product of any finite number of the above sets is considered in the literature.

1.3 Classes of Functions

In this section, we will list the most studied classes in the literature, in different fields of computer science.

Boolean Function Classes: When the range of the function is $R_{n}=\{0,1\}$ we call the function Boolean function. Here we will consider classes $C$ of Boolean functions when the domain is $X_{n}=\{0,1\}^{n}$. For any class defined below when we say that $f$ is $C$, we mean that $f\in C$. Abusing the terminology, every function $f\in C$ is regarded as a representation of the function (formula) and as a function, and we will use both interchangeably.

The most studied classes in the literature are:

Variable (Var): The class Var is the class of functions $\{x_{1},\ldots,x_{n}\}$, where for $a\in\{0,1\}^{n}$, $x_{i}(a)=a_{i}$. We also define Lit$=\{x_{1},\ldots,x_{n}\}\cup\{\bar{x}_{1},\ldots,\bar{x}_{n}\}$ the class of literals. Here $\bar{x}$ is the logic negation of $x$.

Learning the class Var is equivalent to playing the Rényi-Ulam game, [224, 227, 265]. 2. 2.

$d$-Monotone Clause ($d$-MClause) and MClause: The class $d$-MClause is the class of all functions $f_{S}:\{0,1\}^{n}\to\{0,1\}$ where $S\subseteq[n]:=\{1,2,\ldots,n\}$ and $|S|\leq d$ such that $f_{S}(x_{1},\ldots,x_{n})=1$ if and only if $x_{i}=1$ for some $i\in S$. When $S=\emptyset$ then $f_{\emptyset}=0$. Such function can also be expressed as a logic monotone clause $f_{S}=x_{i_{1}}\vee\cdots\vee x_{i_{k}}$ where $S=\{i_{1},\ldots,i_{k}\}$, $k\leq d$ and $\vee$ is the logic “or” function (disjunction). We denote $n$-MClause by MClause.

Learning $d$-MClause is equivalent to group testing, [111, 113, 114]. See many other equivalent problems in [226] and reference within. 3. 3.

$d$-Clause and Clause: The class $d$-Clause is the class of all functions $f_{S,R}:\{0,1\}^{n}\to\{0,1\}$ where $S\cap R=\emptyset$, $S\cup R\subseteq[n]$ and $|S\cup R|\leq d$ such that $f_{S,R}(x_{1},\ldots,x_{n})=1$ if and only if $x_{i}=1$ for some $i\in S$ or $x_{j}=0$ for some $j\in R$. Such function can be expressed as a logic clause $f_{S,R}=x_{i_{1}}\vee\cdots\vee x_{i_{k}}\vee\bar{x}_{j_{1}}\vee\cdots\vee\bar{x}_{j_{r}}$ where $S=\{i_{1},\ldots,i_{k}\}$, $R=\{j_{1},\ldots,j_{r}\}$, and $r+k\leq d$. We denote $n$-Clause by Clause. 4. 4.

$d$-Monotone Term ($d$-MTerm), $d$-Term ($d$-Term), MTerm and Term: The same as the above classes, but replace $\vee$ with the logic “and” function $\wedge$ (Conjunction). The functions in MTerm are sometimes called monomials, and the class MTerm is also denoted by Monomial. That is, a monomial is a conjunction of variables, i.e., $x_{j_{1}}\wedge x_{j_{2}}\wedge\cdots\wedge x_{j_{r}}$ where $1\leq j_{1}<j_{2}<\cdots<j_{r}\leq n$. Here we will sometimes use the arithmetic $\times$ of the field $F_{2}$ for $\wedge$ and write $x_{j_{1}}\wedge x_{j_{2}}\wedge\cdots\wedge x_{j_{r}}$ as $x_{j_{1}}x_{j_{2}}\cdots x_{j_{r}}$.

For a class $C$, the dual class of $C$ is the class

[TABLE]

Obviously, $(C^{D})^{D}=C$, $d$-ClauseD=$d$-Term and $d$-MClauseD=$d$-MTerm. 5. 5.

$d$-XOR and XOR: The same as the $d$-Term class, but replace $\wedge$ with the logic exclusive or function $\oplus$. Here, we will instead use the arithmetic $+$ of the finite field $F_{2}=\{0,1\}$. Since $\bar{x}=x+1$, every function in XOR is of the form $f=x_{i_{1}}+\cdots+x_{i_{k}}+\xi$ where $1\leq i_{1}<i_{2}<\cdots<i_{k}\leq n$ and $\xi\in\{0,1\}$. 6. 6.

$d$-Junta: Let $f:\{0,1\}^{n}\to\{0,1\}$. A variable $x_{i}$ is said to be relevant in $f$ if there are two assignments $a,b\in\{0,1\}^{n}$ such that $a_{i}\not=b_{i}$, for all $j\not=i$ we have $a_{j}=b_{j}$, and $f(a)\not=f(b)$. The class $d$-Junta is the class of all Boolean functions with at most $d$ relevant variable. This function can be represented by a truth table of size $2^{d}$ of all the relevant variables. 7. 7.

$d$-MJunta: For two assignments $a,b\in\{0,1\}^{n}$ we write $a\leq b$ if for every $i$, $a_{i}\leq b_{i}$. A Boolean function $f:\{0,1\}^{n}\to\{0,1\}$ is monotone if for every two assignments $a,b\in\{0,1\}^{n}$, if $a\leq b$ then $f(a)\leq f(b)$. It is easy to see that Monotone functions are closed under disjunction and conjunction. That is, if $f$ and $g$ are monotone functions then $f\wedge g$ and $f\vee g$ are monotone functions.

The class $d$-MJunta is the class of all monotone functions in $d$-Junta. That is, the class of all monotone functions with at most $d$ relevant variables. 8. 8.

Decision Tree (DT): One of the important representations of Boolean functions $f:\{0,1\}^{n}\to\{0,1\}$ is decision tree. A decision tree formula is defined as follows: The constant functions [math] and $1$ are decision trees. If $f_{0}$ and $f_{1}$ are decision trees then, for all $i$,

[TABLE]

is a decision tree (can also be expressed as $f^{\prime}=x_{i}f_{1}\vee\bar{x}_{i}f_{0}$ or $f^{\prime}=x_{i}f_{1}+\bar{x}_{i}f_{0}$). Every decision tree $f^{\prime}$ can be represented as a tree $T(f^{\prime})$. If $f^{\prime}=1$ or [math] then $T(f^{\prime})$ is a node labeled with $1$ or [math], respectively. If $f^{\prime}=$[if $x_{i}=0$ then $f_{0}$ else $f_{1}$], then $T(f^{\prime})$ has a root labeled with $x_{i}$ and has two outgoing edges. The first edge is labeled with [math] and is pointing to the root of $T(f_{0})$ and the second is labeled with $1$ and is pointing to the root of $T(f_{1})$. See Figure 2.

The depth of the decision tree $f^{\prime}$ is the depth of the tree $T(f^{\prime})$. That is the number of edges of the longest path from the root to a leaf in a tree. The size of the decision tree $f^{\prime}$ is the number of leaves in $T(f^{\prime})$, that is, the number nodes in $T(f^{\prime})$ that are labeled with [math] and $1$.

Every Boolean function $f:\{0,1\}^{n}\to\{0,1\}$ can be represented as a DT. The representation is not unique. The following are subclasses of DT.

1. (a)

Depth $d$ Size $s$ Decision Tree (DTd,s): The class ${\rm DT}_{d,s}$ is the class of all decision trees of depth at most $d$ and size at most $s$. 2. (b)

Depth $d$ Decision Tree (DTd): The class ${\rm DT}_{d}$ is the class of all decision trees of depth at most $d$. That is, ${\rm DT}_{d}={\rm DT}_{d,2^{d}}$. 3. (c)

Monotone DT (MDTd,s, MDTd): functions in the above classes that are monotone. 4. (d)

Decision List (DL),[228]: functions $f\in$DT where every internal node in $T(f)$ is pointing to at least one leaf. 5. (e)

Depth $d$-Decision List ($d$-DL): $d$-DL is a decision list of depth at most $d$.

Learning decision tree is equivalent to solving problems in databases, decision table programming, concrete complexity theory, switching theory, pattern recognition, and taxonomy, [206], computer vision, [23].

Disjunctive Normal Form (DNF): A DNF is another important representation of Boolean function $f:\{0,1\}^{n}\to\{0,1\}$. A DNF formula is a formula of the form

[TABLE]

where each $T_{i}\in$Term is a term. The size of $f$ is $s$.

Every Boolean function $f:\{0,1\}^{n}\to\{0,1\}$ can be represented as a DNF. The representation is not unique. It is easy to see that every decision tree of size $s$ can be represented as DNF of size at most $s$.

The subclasses of DNF considered in the literature are

1. (a)

$r$-DNF: The class of DNFs with terms from $r$-Term. 2. (b)

$s$-term DNF: The class of DNFs with at most $s$ terms. 3. (c)

$s$-term $r$-DNF: The class of DNFs with at most $s$ terms each of which is an $r$-Term. 4. (d)

Read-Once $C$: Here $C$ is one of the above classes. Read-Once $C$ is the class of functions $f$ in $C$ where each variable appears at most once in $f$. 5. (e)

Read-Twice, Read-Thrice, Read-$t$ $C$: The class of functions $f$ in $C$ where each variable appears at most twice (resp. three times and $t$ times) in $f$. 10. 10.

Monotone DNF (MDNF): The class MDNF is the class of DNF with monotone terms (i.e., terms in MTerm). Every monotone function (See the definition in item 7) has a monotone DNF representation. This representation is one of the most popular canonical structures for representing Boolean functions. If $f=M_{1}\vee M_{2}\vee\cdots\vee M_{s}$ where each $M_{i}$ is a monomial and no two monomials $M_{i}$ and $M_{j}$, $i\not=j$ satisfies $M_{i}\wedge M_{j}=M_{i}$, then we say that $f$ is a reduced monotone DNF. Every monotone Boolean function $f$ has a unique representation as a reduced monotone DNF [1]. This representation is uniquely determined by the minterms of the function. That is, the assignments $a\in\{0,1\}^{n}$ where $f(a)=1$ and flipping any entry that is $1$ in $a$ to [math] changes the value of the function to zero. Each minterm $a$ of $f$ corresponds, one-to-one, to a monomial $M=\vee_{a_{i}=1}x_{i}$ in the reduced monotone DNF representation of $f$. The following are subclasses of MDNF.

1. (a)

$r$-MDNF: The class of MDNFs with monomials of size at most $r$. That is, terms from $r$-MTerm. 2. (b)

MDNF: The class MDNF is $n$-MDNF. 3. (c)

$s$-term MDNF: The class of MDNFs with at most $s$ monomials. 4. (d)

$s$-term $r$-MDNF: The class MDNFs with at most $s$ monomials of size at most $r$. 5. (e)

Read-Once, Read-Twice, Read-Thrice, Read-$t$ $C$, where $C$ is one of the above classes, is the class of functions $f$ in $C$ where each variable appears at most once (resp. twice, three times and $t$ times) in $f$.

Learning Monotone DNF and subclasses of Monotone DNF equivalent to problems in computational biology that arises in whole-genome shotgun sequencing, [11], and DNA phisical mapping, [144]. 11. 11.

Conjunctive Normal Form (CNF): The class CNF is the dual class (See the definition in item 4) of DNF (where $\wedge$ is replaced with $\vee$ and vice versa). In a similar way as above we define the classes $r$-MCNF, MCNF, $s$-clause CNF, $s$-clause MCNF, $s$-clause $r$-CNF and $s$-clause $r$-MCNF. 12. 12.

CDNF. The class of CDNF is the class of formulas of the form $(f,g)$ where $f$ is a DNF, $g$ is a CNF and $f=g$. The size of $(f,g)$ as $s+t$ where $f$ is $s$-term DNF and $g$ is $t$-clause CNF.

The following are subclasses of CDNF

1. (a)

CDNFs,t. The class of CDNFs,t is the class CDNF, $(f,g)$, where $f$ is a DNF of size at most $s$ and $g$ is a CNF of size at most $t$. 2. (b)

$r$-CDNFs,t: The class of $(f,g)\in$CDNFs,t where $f$ is $r$-DNF of size at most $s$ and $g$ is $r$-CNF of size at most $t$. 3. (c)

$r$-CDNF: The class $\cup_{s,t}$CDNFs,t. 4. (d)

MCDNF: The class of Monotone CDNF. 5. (e)

MCDNFs,t: The class of Monotone CDNFs,t. 6. (f)

$r$-MCDNF: The class of Monotone $r$-CDNF.

Learning CDNF is equivalent to problems in data-mining, graph theory and reasoning and knowledge representation, [118]. 13. 13.

Boolean Multivariate Polynomial (BMP). The class BMP is the class of multivariate polynomials over the binary field $F_{2}$. That is, a function $f:F_{2}^{n}\to F_{2}$ of the form

[TABLE]

where each $M_{i}$ is a monomial. The size of $f$ is $s$.

Every Boolean function $f:F_{2}^{n}\to F_{2}$ can be represented as a BMP. The representation is unique. It is easy to see that every decision tree of size $s$ and depth $t$ can be represented as BMP of size at most $2^{t}s$.

1. (a)

$r$-BMP: The class of BMPs with monomials of size at most $r$, i.e., in $r$-MTerm. This class is also called the class of multivariate polynomial of degree $r$ over $F_{2}$. 2. (b)

$s$-monomial BMP: The class BMPs with at most $s$ monomials. This class is also called the class of sparse multivariate polynomial over $F_{2}$. 3. (c)

$s$-monomial $r$-BMP: The class of BMPs with at most $s$ monomials of size at most $r$. This class is also called the class of sparse multivariate polynomial of degree $r$ over $F_{2}$. 14. 14.

XOR of Terms (XT): The class XT is the class of XOR of terms, $T_{1}+T_{2}+\cdots+T_{s}$ where $T_{i}\in$ Term.

1. (a)

$r$-XT: The class of XTs with terms of size at most $r$. 2. (b)

$s$-term XT: The class of XTs with at most $s$ terms. 3. (c)

$s$-term $r$-XT: The class of XTs with at most $s$ terms of size at most $r$.

Notice that XT with terms from MTerm is BMP. Since every term of size $r$ can be represented as $2^{r}$-monomial $r$-BMP, every $s$-term $r$-XT is $(2^{r}s)$-monomial $r$-BMP. 15. 15.

Deterministic Finite Automaton (DFA),[210]: A DFA is a $5$-tuple $A=(Q,\Sigma,\delta,q_{0},F)$ that can be also represented as a directed graph $G=(V,E)$ with labeled edges where $V=Q$ is a finite set of states (the vertices), and $q_{0}\in Q$ is the start state. $\Sigma$ is a finite set of symbols called the alphabet. $\delta$ is the transition function $\delta:Q\times\Sigma\to Q$. The edge $(v,u)\in E$ in $G$ is labeled with $\sigma\in\Sigma$ if and only if $\delta(v,\sigma)=u$. This transition function defines, for every string $s\in\cup_{i\geq 0}\Sigma^{i}$, a unique path in the graph $q_{0},q_{1},\ldots,q_{|s|}$ (here, $|s|$ is the number of symbols in $s$) that starts from $q_{0}$ and for every $0\leq i\leq|s|-1$, $\delta(q_{i-1},s_{i})=q_{i}$. We denote the final state in this path $q_{|s|}$ as $\delta(q_{0},s)$. The set $F\subset Q$ is the set of accept states.

Every DFA $A$ defines a Boolean function $f:\cup_{i\geq 0}\Sigma^{i}\to\{0,1\}$ where $f(s)=1$ if and only if $\delta(q_{0},s)\in F$. When $\Sigma=\{0,1\}$ then a DFA for the Boolean function $f:\{0,1\}^{n}\to\{0,1\}$ is a DFA such that: for every $a\in\{0,1\}^{n}$ we have $f(a)=1$ if and only if $\delta(q_{0},a)\in F$. 16. 16.

Boolean Multiplicity Automata Function (BMAF),[244]: A Boolean Multiplicity Automata Function is a function of the form:

[TABLE]

where each $A_{i}(x_{i})$ is $s_{i}\times s_{i+1}$ matrix that its entries are Boolean univariate polynomials in $x_{i}$ over $F_{2}$, i.e., $ax_{i}+b$ for $a,b\in F_{2}$, and $s_{1}=s_{n+1}=1$. The size of a BMAF is defined as $\max_{i}s_{i}$.

See [44] for other ways to represent this class. 17. 17.

Boolean Halfspace (Perceptron, Threshold) (BHS): A Boolean Halfspace is a function $f:\{0,1\}^{n}\to\{0,1\}$ of the following form:

[TABLE]

where $w_{1},\ldots,w_{n},u$ are real numbers. The constants $w_{1},\ldots,w_{n}$ are called the weights of the Halfspace, and $u$ is called the threshold. For $W\subseteq\Re$ we define

1. (a)

BHS$(W)$: The class of Boolean Halfspaces with weights $w_{i}\in W$. 2. (b)

$d$-BHS$(W)$: The class of functions in BHS$(W)$ with at most $d$ relevant variables. 18. 18.

Boolean Circuit (BC) and Boolean Formula (BF) A Boolean circuit over the set of variables $x_{1},\ldots,x_{n}$ is a directed acyclic graph where every node in it with indegree zero is called an input gate and is labeled by either a variable $x_{i}$ or a Boolean constant $\{0,1\}$. Every other gate is either a node with indegree one and is labeled $\neg$ (unary NOT) or a node with indegree two and is labeled by either, $\wedge$ (binary AND) or $\vee$ (binary OR). A Boolean formula is a circuit in which every gate has outdegree one.

The size of a Boolean circuit is the number of gates in it, and its depth is the length of the longest directed path in it.

1. (a)

Monotone Boolean Circuit (MBC) and Monotone Boolean Formula (MBF) MBC and MBF are Boolean circuit and Boolean formula, respectively, with no $\neg$ gate. 2. (b)

Read Once Formula (ROF). The class of Boolean read-once formula. A Boolean read-once formula is a formula such that every input variable $x_{i}$ appears in at most one input gate. 3. (c)

Monotone Read Once Formula (MROF). The class of monotone read-once formula. 4. (d)

Read-Once, Read-Twice, Read-Thrice, Read-$t$ $C$, where $C$ is one of the above classes, is the class of functions $f$ in $C$ where each variable appears at most once (resp. twice, three times and $t$ times) in $f$.

See other classes in [1, 2, 17, 26, 33, 34, 58, 73, 92, 93, 110, 118, 132, 158, 251].

Here are relations between some of the classes mentioned above.

[TABLE]

For two classes $C_{1}$ and $C_{2}$ we write $C_{1}\subseteq C_{2}$ (written as $C_{1}\to C_{2}$ in the above diagram) if every function in $C_{1}$ of size $s$ is equivalent to a function in $C_{2}$ of size $O(s)$.

As for functions that are not Boolean, the literature is poor in studying the exact learnability of classes of functions with finite discrete domain or/and range from membership queries only. On the other hand, there is a substantial body of literature on learning and testing arithmetic classes.

We now give some of the arithmetic classes defined in the literature

Arithmetic Classes: Arithmetic classes represent function $f:X\to R$ where $R$ is an algebraic structure such as field or ring. For exact learning, the most investigated arithmetic classes in the literature are

$(r,V)$-Linear Functions ($(r,V)$-LF), where $r$ is an integer, $V\subset\Re$ and $\Re$ is the set of real numbers. An $(r,V)$-LF is a function $f:\{0,1\}^{n}\to\Re$ of the form $v_{1}x_{i_{1}}+\cdots+v_{r^{\prime}}x_{i_{r^{\prime}}}$ where $r^{\prime}\leq r$ and $v_{i}\in V$ for all $i=1,\ldots,r^{\prime}$. The class $r$-LF is the class $(r,\{0,1\})$-LF and LF is the class $n$-LF.

Learning $(r,V)$-LF is equivalent to coin weighing problem [37] and signature coding problem [50]. 2. 2.

$(r,V)$-Quadratic Functions ($(r,V)$-QF), where $r$ is an integer and $V\subset\Re$. A $(r,V)$-QF is a function $f:\{0,1\}^{n}\to\Re$ of the form $x^{T}Ax$ where $x\in\{0,1\}^{n}$ and $A$ is a symmetric $n\times n$ matrix with at most $r$ non-zero entries from $V$. The class $r$-QF is the class $(r,\{0,1\})$-QF.

Learning $(r,V)$-QF is equivalent to problems in molecular biology [55]. 3. 3.

Multivariate Polynomial (MP): Let $F$ be a field. A multivariate polynomial over $F$ is a function $f:F^{n}\to F$ of the form

[TABLE]

where $I\subseteq N^{n}$, $N=\{0,1,2,\cdots\}$ and $a_{i}\in F$. The size of $f$ is $|f|:=|I|$. The term $x_{1}^{i_{1}}\cdots x_{n}^{i_{n}}$ is called monomial. The monomial is called $t$-monomial if $|\{j\ |\ i_{j}\not=0\}|\leq t$. The multivariate polynomial is said to be of degree $d$ if $i_{1}+\cdots+i_{n}\leq d$ for all $i\in I$, $s$-sparse if $|I|\leq s$ and with $t$-monomials if all its monomials are $t$-monomials.

When the field $F$ is finite then every function $f:F^{n}\to F$ can be represented as a multivariate polynomial. This fact is not true for infinite fields. 4. 4.

Multiplicity Automata Function: A Multiplicity Automata Function (MAF) over the field $F$ is a function of the form

[TABLE]

where each $A_{i}(x_{i})$ is $s_{i}\times s_{i+1}$ matrix that its entries are linear functions in $(x_{1},\ldots,x_{n})$ (i.e., $\sum_{i}a_{i}x_{i}+b$ where $a_{i},b\in\Re$) and $s_{1}=s_{n+1}=1$. The size of a MAF $f$ is $\max_{i}s_{i}$.

This class contains the class MP in a sense that every MP of size $s$ has a MAF of size $s$.

See [44] for other representations of MAF. 5. 5.

Arithmetic Circuit (AC) and Arithmetic Formula (AF) An arithmetic circuit over the field $F$ and the set of variables $x_{1},\ldots,x_{n}$ is a directed acyclic graph where every node in it with indegree zero is called an input gate and is labeled by either a variable $x_{i}$ or a field element. Every other gate is labeled by either $+$ or $\times$, in the first case, it is a sum gate and in the second a product gate. An arithmetic formula is a circuit in which every gate has outdegree one.

The size of a circuit is the number of gates in it, and its depth is the length of the longest directed path in it. The degree of a circuit is equal to the degree of the polynomial output by the circuit. 6. 6.

Arithmetic Read-Once Formula (AROF). An arithmetic read-once formula is a formula such that every input variable $x_{i}$ appears in at most one input gate.

Here are relations between some of the classes we’ve defined

[TABLE]

See other classes in [43, 239, 240, 255] and references therein.

1.4 Learning Algorithms and Complexity

The learning algorithm can be sequential or parallel, deterministic or randomized and adaptive (AD), $r$-round ($r$-RAD) or non-adaptive (NAD).

In the adaptive algorithm, the queries can depend on the answers to the previous ones. In the non-adaptive algorithm they are independent of the previous one and; therefore, one can ask all the queries in one parallel step. We say that an adaptive algorithm is $r$-round adaptive ($r$-RAD) if it runs in $r$ stages where each stage is non-adaptive. That is, the queries may depend on the answers to the queries in the previous stages but independent on the answers to the queries of the current stage.

The randomized algorithm can be either Monte Carlo (MC) or Las Vegas (LV). A Monte Carlo algorithm is a randomized algorithm whose running time is deterministic, but whose output may be incorrect with probability at most $\delta$. A Las Vegas algorithm is a randomized algorithm that always gives a correct hypothesis. That is, it always produces a hypothesis that is equivalent to the target function. The complexity of a Las Vegas algorithm is measured by the expected running time, the expected number of queries and the expected number of rounds.

The goal is to ask the minimum number of queries and minimize the running time and space complexity of the algorithm and/or other resources such as the number of processors (for parallel algorithms) or the number of random bits (for randomized algorithms).

1.5 Polynomially, Efficiently and Optimally Learnable

In this subsection and the next, we try to unify the different definitions used in the literature of the efficiency of the query complexity and time complexity of exact learning algorithms. We will use the following new terminologies defined below: “learnable”, “polynomially learnable” “efficiently learnable”, “almost optimally learnable” and “optimally learnable”.

Let $C$ be a class of functions. Let ${\rm OPT}_{A}(C)$ be the minimum number of membership queries that a learner, with unlimited computational power, needs to learn $C$ with algorithms of type $A$. The algorithm type, $A$, can be adaptive (AD), non-adaptive (NAD) or $r$-round ($r$-RAD). For example, we will use ${\rm OPT}_{\rm AD}$ for the adaptive algorithm and ${\rm OPT}_{\rm NAD}$ for the non-adaptive algorithm. When the algorithm is randomized we also add, as a subscript, MC for Monte Carlo algorithms and LV for Las Vegas algorithms.

In complexity theory, a polynomial time algorithm is an algorithm that runs in polynomial time in the input size. In the exact learning model, the time complexity of learning the class $C$ is, at least, the query complexity, ${\rm OPT}_{A}(C)$, which can be exponential in the target function size. Therefore, polynomial time learning algorithm for $C$ will be defined as a learning algorithm that asks $poly({\rm OPT}_{A}(C),$ $n)$ queries and runs in time $poly({\rm OPT}_{A}(C),n)$, where $n$ is the size of the elements in the domain $X$. Such classes are called polynomially learnable or just learnable classes. This is the definition used in the literature for learnability of classes.

Since the time complexity of any learning algorithm for $C$ is at least $n\cdot{\rm OPT}_{A}(C)$ we may say that learning algorithms that run in time $poly({\rm OPT}_{A}(C),$ $n)$ are “efficient algorithms” in time. However, this is not true for the query complexity. We will argue here, by the following example, that the above definition of $poly({\rm OPT}_{A}(C),n)$ for the query complexity is not the best definition for query-efficiency of exact learning from membership queries.

Take for example the class $C=d$-MClause. We will show in Subsection 4.5 that ${\rm OPT}_{{\rm AD}}(C)=\Theta(d\log n)$. Therefore, one would expect that a query-efficient learning algorithm for this class asks $poly(d,\log n)$ queries and not $poly(d\log n,n)=poly(n)$ queries as defined above. The time complexity cannot be less than $n\cdot{\rm OPT}_{A}(C)$, so the definition of $poly({\rm OPT}_{A}(C),n)$ in the time complexity is passable.

Therefore, we will suggest the following definition for efficient learning. If the algorithm for learning $C$ asks $poly({\rm OPT}_{A}(C))$ queries (rather than $poly({\rm OPT}_{A}(C)$ $,n)$) and runs in time $poly({\rm OPT}_{A}(C),n)$, then we call the class efficiently learnable111We will not use the term “polynomially learnable” for this case to avoid confusion with the definition in the literature..

Another concern with this new definition is that in many areas, (such as combinatorial group testing and game theory) membership query is considered to be very costly. Therefore, one must find polynomial time learning algorithms that ask a minimum number of queries. Therefore, we will introduce here two other definitions: If there is a learning algorithm for $C$ that asks ${\rm OPT}_{A}(C$ $)^{1+o(1)}$ queries and runs in time $poly({\rm OPT}_{A}(C),n)$, then we call the class almost optimally learnable. If there is a learning algorithm for $C$ that asks $O({\rm OPT}_{A}(C))$ queries and runs in time $poly({\rm OPT}_{A}(C)$ $,n)$, then we call the class optimally learnable.

In many cases, the query complexity ${\rm OPT}_{A}(C)$ is a function of several parameters that are related to the class $C$. For example, the query complexity $\Theta(d\log n)$ of $d$-MClause also depends on $d$. We say that the query complexity of a learning algorithm is optimal (resp. almost optimal, efficient or polynomial) in some parameter if assuming the other parameters are constant, the query complexity of the algorithm is optimal (resp. almost optimal, efficient or polynomial). So a learning algorithm for $d$-MClause that asks $d\cdot poly(\log n)$ queries is efficient, optimal in $d$ and efficient in $n$.

We say that the class $C$ is query-polynomially (resp. query-efficiently, almost query-optimally or query-optimally) learnable in time $T$ if the number of queries is as above (for polynomially, efficiently, almost optimally and optimally, respectively) but the time complexity is $T$.

We summarize all the above definitions in the following table:

Terminology

Query Complexity

Time Complexity

Polynomially Learnable

or Learnable

${\rm poly}({\rm OPT}_{A}(C),n)$

${\rm poly}({\rm OPT}_{A}(C),n)$

Efficiently Learnable

${\rm poly}({\rm OPT}_{A}(C))$

${\rm poly}({\rm OPT}_{A}(C),n)$

Almost Optimally Learnable

${\rm OPT}_{A}(C)^{1+o(1)}$

${\rm poly}({\rm OPT}_{A}(C),n)$

Optimally Learnable

$O({\rm OPT}_{A}(C))$

${\rm poly}({\rm OPT}_{A}(C),n)$

Optimally Learnable in $n$

$O({\rm OPT}_{A}(C))$

${\rm poly}({\rm OPT}_{A}(C),n)$

when the other parameters are constant

Query-Optimally Learnable

$O({\rm OPT}_{A}(C))$

$T(n)$

in time $T(n)$

1.6 Strongly Polynomially, Efficiently and Optimally Learnable

Let $C$ be a class of functions. Suppose there is an integer parameter $r$ and classes $C_{r}$ such that $C=\cup_{r\geq 0}C_{r}$. We say that $C$ is strongly $r$-polynomially learnable if there is a learning algorithm for $C$ such that, for every target function $f\in C_{r}$, the algorithm runs in time $poly({\rm OPT}(C_{r}),n)$ and asks at most $poly({\rm OPT}(C_{r}),n)$ queries. In the same way as in the above subsection we define strongly $r$-efficiently learnable, strongly almost $r$-optimally learnable, strongly $r$-optimally learnable and $r$* learnable in time $T$*.

For example, it is known that ${\rm OPT}_{{\rm AD}}(d$-MClause$)=\Theta(d\log n)$. Obviously, MClause$=\cup_{d\geq 0}d$-MClause. For $f\in$MClause, let $d(f)$ be the minimum integer such that $f\in d(f)$-MClause. That is, $d(f)$ is the number of relevant variables of $f$. The class MClause is adaptively strongly $d$-optimally learnable if there is an adaptive algorithm that with a target function $f$ that is MClause, the algorithm runs in time $poly(n)$ and asks $\Theta(d(f)\log n)$ queries. That is, the algorithm runs in time $\Theta(d\log n)$ even when $d$ is not known to the learner.

Recall that, for a function $f:\{0,1\}^{n}\to\{0,1\}$, we say that $x_{i}$ is relevant in $f$ if there is an assignment $a$ such that $f(a)\not=f(a+e_{i})$, where $\{e_{j}\}_{j}$ is the standard bases, and $+$ is the bitwise exclusive or. That is, if $f$ depends on the variable $x_{i}$. We say that $x_{i}$ is irrelevant in $f$ if it is not relevant in $f$. For a class $C$, let $C_{d}$ be the set of all functions in $C$ that have at most $d$ relevant variables. Then $C=\cup_{d\geq 0}C_{d}$. A strongly $d$-efficiently learnable class $C$ will be called strongly attribute-efficiently learnable. The same definition applies for attribute-polynomially, almost attribute-optimally, attribute-optimally and attribute learnable in time $T$.

The definition of “strongly attribute-efficient” in [62] is equivalent to our definition of “strongly attribute-optimally learnable in $n$”.

1.7 Testing Problems

The following problems are also considered in the literature

Equivalence Testing: Given two teachers where each one has a function from $C$. The learner can ask each one membership queries. Test whether the two functions are equivalent. The minimum number of queries is denoted by ${\rm OPT}^{{\rm ET}}_{A}$ $(C)$.

For the non-adaptive algorithms, this is equivalent to constructing a set of assignments $A$ such that for every $f,g\in C$, $f\not=g$ there is $a\in A$ such that $f(a)\not=g(a)$. Such a set is called an equivalent test set or universal identification sequence [146]. Obviously, for deterministic algorithms,

[TABLE]

Also, it is easy to show that.

[TABLE]

We give a proof sketch of the latter for completeness

Proof

Let ${\cal A}$ be an adaptive algorithm that learns $C$. We run ${\cal A}$. Each time it asks a membership query $a$, we ask both teachers that membership query. Since ${\cal A}$ learns $C$, for some assignment we get different answers. Therefore ${\rm OPT}_{{\rm AD}}^{{\rm ET}}(C)\leq 2\cdot{\rm OPT}_{\rm AD}(C).$

Let ${\cal B}$ be an adaptive algorithm for identity testing the class $C$. Let $T$ be the teacher with the target function $f\in C$. We run ${\cal B}$ with $T$ and a dummy teacher $T^{\prime}$ that always gives the same answer as $T$ as long as there is a function $g\in C\backslash\{f\}$ that is consistent with $f$ on all the answers to the queries. Finally, no other function is consistent with $f$ and $f$ is uniquely determined. Therefore ${\rm OPT}_{{\rm AD}}^{{\rm ET}}(C)\geq 2\cdot{\rm OPT}_{\rm AD}(C).$ ∎ 2. 2.

Identity Testing from $H$ and Teaching Dimension: Given a teacher that has a function $f$ from $C$. Given a function $h\in H\supseteq C$. The learner has $h$ and can ask the teacher membership queries. Test whether $f=h$.

A non-adaptive algorithm is equivalent to constructing, for every $h\in H$, a set of assignments $A_{h}$ such that for every $f\in C$, $f\not=h$ there is $a\in A_{h}$ such that $f(a)\not=h(a)$. Such a set is called an identity test set for $h$ with respect to $C$.

Notice that if $h\in C$ then an identity testing set for $h$ uniquely determines $h$. If $h\not\in C$ then an identity testing set for $h$ gives a proof that $h$ does not belong to $C$. Therefore, we will also call this set membership test set.

The maximum, over all the functions $h\in H$, of the minimum size identity test set for $h$ is denoted by ${\rm TD}(H,C)$ or ${\rm OPT}^{{\rm IT}}_{\rm NAD}(H,C)$. When $H=C$ the set $A_{h}$ is called a teaching set, and ${\rm TD}(C,C)$ is denoted by ${\rm TD}(C)$ or ${\rm OPT}^{{\rm IT}}_{\rm NAD}(C)$ and is called the teaching dimension of the concept $C$, [142, 151, 249].

We have ${\rm OPT}_{\rm AD}(C)\geq{\rm TD}(C)$. To show this fact, suppose ${\cal A}$ is an algorithm that adaptively learns $C$. Run the algorithm with the target $h$. The set of all the queries asked is a teaching set for $h$.

For studies in teaching dimension see the following and reference therein [142, 151, 157, 181, 216, 233, 242, 247, 249]. 3. 3.

Constant Testing: Given a teacher that has a function $f$ from $C$. Test, with membership queries, whether $f$ is not a constant function. This is equivalent to constructing a set of assignments $A$ such that, for every non-constant function $f\in C$, there is $a,b\in A$ such that $f(a)\not=f(b)$. Such a set is called a constant test set for $C$. ${\rm OPT}^{{\rm CS}}(C)$ denotes the minimum size constant test set for $C$. 4. 4.

Zero Testing: Given a teacher that has a function $f\not=0$ from $C$. Test whether $f=0$ with membership queries. This is equivalent to constructing a set of assignments $A$ such that, for every non-zero function $f\in C$, there is $a\in A$ such that $f(a)\not=0$. Such a set is called a zero test set or a hitting set for $C$. The minimum size hitting set for $C$ is denoted by ${\rm OPT}^{{\rm HS}}(C)$. It is easy to show that

[TABLE]

and

[TABLE]

where $C-h=\{f-h\ |\ f\in C\}$ and $C-C=\{f-g\ |\ f,g\in C\}$. For Boolean functions, $f$ and $g$, $f-g=f\oplus g$ where $\oplus$ is the exclusive or operation.

See other results in the above references and the references therein.

2 Bounds on OPT for Boolean Functions and Algorithms

In this section, we give some bounds for ${\rm OPT}_{A}(C)$ for classes of Boolean functions $C$ and some exponential time algorithms that are query-efficient for any class $C$.

2.1 OPT for Adaptive Algorithms

We first state the following information-theoretic lower bound for deterministic learning algorithm. Throughout the paper, we write $\log x$ for $\log_{2}x$.

Lemma 1

Let $C$ be a class of Boolean functions. Any deterministic learning algorithm for $C$ must ask at least $\log|C|$ membership queries. That is

[TABLE]

In fact, the bound is also true for Monte Carlo and Las Vegas algorithms. See for example [12]

Lemma 2

Let $C$ be any class of Boolean functions. Any Monte Carlo (and therefore, Las Vegas) randomized learning algorithm that learns $C$ with probability at least $3/4$ must ask at least $\log|C|-1$ membership queries. That is

[TABLE]

We now give upper and lower bounds for ${\rm OPT}_{A}(C)$ using the following combinatorial measure that is defined in [158, 208]. Let $C$ be a class of Boolean functions $f:X\to\{0,1\}$. Let $h:X\to\{0,1\}$ be any function. We say that a set $T_{h}\subseteq X$ is a specifying set for $h$ with respect to $C$ if

[TABLE]

That is, there is at most one concept in $C$ that is consistent with $h$ on $T_{h}$. Denote $T(C,h)$ the minimum size of a specifying set for $h$ with respect to $C$. The extended teaching dimension of $C$ is

[TABLE]

Let $h:X\to\{0,1\}$ be any function. For every $f\in C$, $f\not=h$ there is an assignment $x_{f}\in X$ such that $f(x_{f})\not=h(x_{f})$. Therefore $\{x_{f}\ |\ f\in C\backslash\{h\}\}$ is a specifying set for $h$ and $T(C,h)\leq|C|$. Therefore

[TABLE]

Notice that for every Boolean function $h\not\in C$ if $T_{h}$ is a specifying set for $h$ and $f\in C$ is a function that is consistent with $h$ on $T_{h}$ then adding an assignment $a$ to $T_{h}$ where $f(a)\not=h(a)$ gives an identity testing set for $h$ with respect to $C$. Therefore,

[TABLE]

In [158, 208], Moshkov proves the following bounds. Here, we will give another proof that gives, asymptotically, the same upper bound for ${\rm OPT}_{\rm AD}(C)$.

Lemma 3

[158, 208]** Let $C$ be any class of Boolean functions. Then

[TABLE]

and

[TABLE]

Proof

The second inequality in (4) follows from (3).

Consider the following algorithm. After the $i$th query, the algorithm defines a set $C_{i}\subseteq C$ of all the functions that are consistent with the membership queries that were asked so far. Consider any $0<\epsilon<1/2$. Now the algorithm searches for an assignment $a\in X$ such that

[TABLE]

If such $a\in X$ exists, then it asks the membership queries ${\rm MQ}(a)$. Define $C_{i+1}=\{f\in C_{i}\ |\ f(a)={\rm MQ}(a)\}$. Obviously, in that case,

[TABLE]

If no such $a\in X$ exists, then the algorithm finds a specifying set $T_{h}$ for $h:={\rm Majority}(C_{i})$, where “Majority” is the majority function. It then asks membership queries for all the assignments in $T_{h}$. If the answers are consistent with $h$ on $T_{h}$, then there is a unique concept $c\in C_{i}$ consistent with the answers, and the algorithm outputs this concept. Otherwise, there is $a\in T_{h}$ such that ${\rm MQ}(a)\not=h(a)$. It is easy to see that in that case

[TABLE]

Denote by $k$ the number of times when the algorithm is left without such an $a\in X$. Then the number rounds when it does find such an $a\in X$ is

[TABLE]

and so the number of queries is upper bounded by

[TABLE]

When $\epsilon$ is chosen so that the last term becomes [math], then $\epsilon$ is roughly $\log{\rm ETD}(C)/$ ${\rm ETD}(C)$. In case of this $\epsilon$, (5) becomes

[TABLE]

In [158, 208], Moshkov gives, for any two integers $t$ and $\ell$, an example of a class $C_{t,\ell}$ where

[TABLE]

So the upper bound in the above lemma is the best possible.

See also the other dimensions, bounds, and techniques in [1, 2, 4, 19, 156, 213].

2.2 Constructing Adaptive Algorithms

Given a class of Boolean functions $C$ as an input. Can one construct an algorithm that learns $C$ with ${\rm OPT}_{{\rm AD}}(C)$ queries? Obviously, with unlimited computational power, this can be done so the question is: How close to ${\rm OPT}_{{\rm AD}}(C)$ can one get when polynomial time (or any other time) is allowed for the construction?

An exponential time algorithm follows from the following

[TABLE]

where $C_{x,\xi}=\{f\in C\ |\ f(x)=\xi\}$. See also [18, 135]. Can one do it in $poly(|C|,|X|)$ time? Hyafil and Rivest, [168], show that the problem of finding ${\rm OPT}_{{\rm AD}}$ is NP-Complete. Laber and Nogueira, [202], show that this problem does not admit an $o(\log|C|)$-approximation unless P=NP. The reduction of Laber and Nogueira, [202], of set cover to this problem with the inapproximability result of Dinur and Steurer [121] for set cover implies that it cannot be approximated to $(1-o(1))\cdot\ln|C|$ in polynomial time unless P=NP.

The query complexity of the algorithm in Lemma 3 is within a factor of

[TABLE]

from OPT. However, unfortunately, the problem of finding a minimum size specifying set for $h$ is NP-Hard, [9, 142, 249].

In [23], Arkin et al. gives the following algorithm: Let $C_{i}$ be the set of functions that are consistent with the answers to the first $i$ membership queries. The $i+1$th membership query of the algorithm is an assignment $a$ that partitions the set $C_{i}$ as evenly as possible, that is, an assignment $a$ that maximizes $\min(|\{f\in C_{i}|f(a)=0\}|,|\{f\in C_{i}|f(a)=1\}|)$. Arkin et al. show in [23] that the query complexity of this algorithm is within a factor of $c\ln|C|$ from ${\rm OPT}$ for some $c>1$. Moshkov in [209] gives the exact ratio of $\ln|C|$. This algorithm runs in time $poly(|C|,|X|)$. In particular, we have the following result. Here, we will give a very simple proof

Lemma 4

[23]**. There is a learning algorithm that runs in time $poly(|C|,|X|)$ and learns $C$ with at most

[TABLE]

queries.

In particular, any class $C$ is adaptively query-efficiently learnable in time poly$(|C|,|X|)$.

Proof

First, we have

[TABLE]

where $C_{x,\xi}=\{f\in C\ |\ f(x)=\xi\}$. This follows from the fact that every membership query can eliminate at most $\max_{a}\min(|C_{a,0}|,|C_{a,1}|)$ functions from $C$. Therefore, there is $b\in X$ such that

[TABLE]

Thus, using $b$ as the first query, eliminates at least $|C|/{\rm OPT}_{{\rm AD}}(C)$ functions from $C$. After the $\ell$th query the number of functions that remain (consistent with the answers to the membership queries) is at most

[TABLE]

So the number of queries of the algorithm is at most $\ell=(\ln|C|){\rm OPT}_{{\rm AD}}(C)$. By Lemma 1, $\ell\leq 0.693\cdot{\rm OPT}_{{\rm AD}}(C)^{2}$.∎

Bshouty et al., [47], show that using the NP-oracle all Boolean classes are efficiently learnable in randomized expected polynomial time. See other results in [47].

2.3 OPT for Non-Adaptive Algorithms

In this subsection, we give some bounds for non-adaptive learning classes Boolean functions. The following result follows from (1) and (2)

Lemma 5

We have

[TABLE] 2. 2.

${\rm OPT}_{{\rm NAD}}(C)$* is equal to half the minimum size equivalent test set for $C$.* 3. 3.

The set $A$ is an equivalent test set for $C$ if and only if $A$ is a hitting set for $C\oplus C:=\{h\oplus g\ |\ h,g\in C\}$. That is,

[TABLE]

We now prove

Lemma 6

[142, 176]**. There is a learning algorithm that runs in time $poly(|X|,$ $|C|)$ and finds a hitting set for $C$ of size at most

[TABLE]

In particular,

There is a non-adaptive learning algorithm that runs in time $poly(|X|,|C|)$ and learns $C$ using at most

[TABLE]

queries. 2. 2.

Any class $C$ is non-adaptively query-efficiently learnable in time poly$(|C|,|X|)$.

Proof

Define for every $x\in X$ the set $C_{x}$ of all the functions $f\in C$ such that $f(x)=1$. Now the hitting set problem is equivalent to the set cover problem, i.e., find the minimal number of elements $S\subset X$ such that $\cup_{x\in S}C_{x}=C$. It is known that the greedy algorithm that at each stage, chooses the set that contains the largest number of uncovered elements, achieves an approximation ratio of $\ln|C|$, [84].

Now (8) follows from (7), Lemma 1 and 3 in Lemma 5.∎

The above reduction shows that the problem of finding a small hitting set is equivalent to finding a small set cover, and therefore, the minimum hitting set problem cannot be approximated in polynomial time to within of factor of less than $(1-o(1))\ln|C|$, [24, 121, 129, 234].

A Hitting set for $C\oplus C$ is also a hitting set for $C$ except probably one function. This follows from the fact that if $A$ is a hitting set for $C\oplus C$ then for each two distinct functions $f,g\in C$ there is $a\in A$ such that $f(a)\not=g(a)$ and therefore $a$ hits one of them. This implies that there is no learning algorithm that runs in time $poly(|X|,$ $|C|)$ and non-adaptively learns $C$ with less than $(\ln|C|){\rm OPT}_{\rm NAD}(C)$ unless P$=$NP.

2.4 OPT for Classes of Small VC-dimension

We have seen in the previous subsection that the query complexity of non-adaptive learning the class $C$ is equal to the minimum size hitting set for $C\oplus C$ and finding a small hitting set is equivalent to finding a small set cover. We now give another way to construct a small hitting set for classes with small Vapnik-Chervonenkis (${\rm VCdim}$) dimension. We first define the ${\rm VCdim}$ of a class $C$.

For a class $C$ and a set $S\subseteq X$, we say that $S$ is shattered by $C$ if for any $T\subseteq S$ there is a function $f\in C$ such that for all $a\in T$ we have $f(a)=0$ and for all $a\in S\backslash T$ we have $f(a)=1$. The ${\rm VCdim}$ of a class $C$, ${\rm VCdim}(C)$, is the maximum integer $d$ such that there is a set $S$ of size $d$ that is shattered by $C$.

Another way to construct a hitting set for $C$ is by choosing a distribution $P$ on the domain $X$ and then repeatedly chooses elements $x\in X$ according to the distribution $P$ until we get a hitting set. This approach can also be used to prove an upper bound for the hitting set size.

Lemma 7

Suppose one can define a distribution $P$ over $X$ such that for every $f\in C$, $\Pr_{P}[f(x)\not=0]\geq\epsilon_{P}$. Then a set of

[TABLE]

elements chosen according to the distribution $P$ is a hitting set for $C$ with probability at least $1/2$.

In particular,

[TABLE]

Proof

The result follows from the $\epsilon$-net theorem [169]. See also Chapter 13 in [28]. ∎

In particular, this also gives the following upper bound for ${\rm OPT}_{{\rm NAD}}$

Lemma 8

Suppose one can define a distribution $P$ over $X$ such that for every $f,g\in C$ we have $\Pr_{P}[f(x)\not=g(x)]\geq\epsilon$. Then a set of $T_{P}(C\oplus C)$ elements chosen according to the distribution $P$ is an equivalent test set for $C$ with probability at least $1/2$.

In particular,

[TABLE]

Brönnimann and Goodrich, [51], show that there is an algorithm that runs in time $poly(|X|,|C|)$ and finds a hitting set for $C$ of size at most

[TABLE]

See also [128].

3 Reductions

In this section, we give some reductions that change an existing algorithm to an algorithm with a better query complexity and an algorithm in another learning model to an algorithm that learns from membership queries. We will show in the sequel how to apply those reductions.

3.1 Reductions for Adaptive Algorithms

In this subsection, we show some reductions from one exact adaptive learning algorithm to another one. Those reductions change the query complexity to be optimal in some of the parameters of the class.

For a class $C$ of functions $f:\{0,1\}^{n}\to\{0,1\}$ we say that $C$ is projection closed if for any $f\in C$, $1\leq i\leq n$ and $\xi\in\{0,1\}$ we have $f|_{x_{i}\leftarrow\xi}\in C$. That is, projecting any variable $x_{i}$ to any value $\xi\in\{0,1\}$ keeps the function in the class $C$. We say that the class $C$ is embedding closed if for any $f\in C$ and any $1\leq k,i\leq n$ we have $f|_{x_{i}\leftarrow x_{k}}\in C$. We note here that almost all the classes considered in the literature are projection and embedding closed.

Here we use the parameter $r$ for the number of relevant variables. For class $C$, the class $C_{r}$ contains all the functions in $C$ with at most $r$ relevant variables. All the classes below are projection closed. It is easy to show that for projection closed class $C$, $C_{r}\supseteq C_{1}$ and $C_{1}$ contains at least $n$ literals. Therefore, ${\rm OPT}_{\rm AD}(C_{r})\geq\log n$.

In [62], Blum et al. show

Lemma 9

[62]** Let $C$ be a class that is projection closed. If $C$ is adaptively learnable in time $T(n)$ with $Q(n)$ queries and there is a constant testing set for $C_{r}$ of size $P(n,r)$ that can be constructed in time $t(n)$ then $C_{r}$ is adaptively learnable in time $poly(n,T(r),Q(r),t(n),P(n,r))$ with

[TABLE]

queries.

In particular, if $P(n,r)=O(\log n)$ (when the other parameters of the class $C$ are constants), then $C_{r}$ is optimally learnable in $n$.

The results in [57] show that if the class is also embedding closed then $P(n,r)=O(r^{6}\cdot Q(r^{2})\log n)$. Therefore, with Lemma 9, we get the query complexity

[TABLE]

When the class is also embedding closed, Bshouty and Hellerstein,[57], show

Lemma 10

[41, 57]** Let $C$ be a projection and embedding closed class. If $C$ is adaptively learnable in time $T(n)$ with $Q(n)$ queries, then $C_{r}$ is adaptively learnable in time $poly(T(n))$ with

[TABLE]

queries.

In particular, if $C$ is learnable then $C_{r}$ is optimally learnable in $n$.

For the randomized algorithm, we have

Lemma 11

[57]** Let $C$ be a projection and embedding closed class. If $C$ is adaptively learnable with a Monte Carlo algorithm in time $T(n)$ with $Q(n)$ queries then $C_{r}$ is adaptively learnable with a Mote Carlo algorithm in time $poly(T(n))$ with

[TABLE]

queries.

3.2 Reductions for Strong Adaptive Algorithms

In some cases, one can achieve strong learning. The following result is from [62]. See the definition of strong learning in Subsection 1.6.

Lemma 12

[62]** Let $C$ be a class that is projection closed. If $C$ is adaptively learnable in time $T(n)$ with $Q(n)$ queries and there is a constant testing set for $C$ of size $P(n)$ that can be constructed in time $t(n)$ then $C$ is adaptively strongly-attribute learnable in time $poly(n,T(r),Q(r),t(n),P(n))$ with

[TABLE]

queries where $r$ is the number of relevant variables of the target.

In particular, if $P(n)=O(\log n)$ (when the other parameters of the class $C$ are constants), then $C$ is strongly attribute-optimally learnable in $n$.

Since $\{0^{n},1^{n}\}$ (the all zero and all one assignments) is a constant testing set for all the monotone functions, $P(n)=2$, in particular, we have

Lemma 13

[62]** Let $C$ be a projection closed class that contains monotone functions. If $C$ is adaptively learnable in time $T(n)$ with $Q(n)$ queries, then $C$ is adaptively strongly attribute learnable in time $poly(n,T(r),Q(r))$ with

[TABLE]

queries.

In particular, $C$ is strongly attribute-optimally learnable in $n$.

So with the above results, one can change algorithms that learn $C$ to algorithms that optimally and strongly attribute-optimally learn $C$ in $n$.

3.3 Reductions for Non-Adaptive Algorithms

In this subsection, we give a reduction from one exact non-adaptive learning algorithm to another one that changes the query complexity to be optimal in some of the parameters.

In [13], Abasi et al. gave the following reduction. We sketch the proof for completeness

Lemma 14

Let $C$ be an embedding closed class such that $|C_{r}|=\Omega(n)$. Let $H$ be a class of Boolean functions and suppose there is an algorithm that, for input $h\in H$, finds the relevant variables of $h$ in time $R(n)$.

If $C$ is non-adaptively learnable from $H$ (the output hypothesis is from $H$) in time $T(n)$ with $Q(n)$ membership queries then $C_{r}$ is non-adaptively learnable from $H$ in time $poly(n,T(q),R(q))$ with

[TABLE]

membership queries where $q\geq 2r^{2}$ is any integer.

In particular, $C_{r}$ is non-adaptively optimally learnable in $n$.

Proof

We use a perfect hash family $P$ that map the variables $X=\{x_{1},\ldots,x_{n}\}$ to a new set of variables $Y=\{y_{1},\ldots,y_{q}\}$ where $q>r^{2}$. This family contains $O(r^{2}/\log(q/r^{2})\log n)$ hash functions and ensures that for almost all the hash functions, different relevant variables of the target are mapped to different variables in $Y$. It also insures that for every non-relevant variable $x_{i}$, almost all the hash functions map the relevant variables and $x_{i}$ to different variables in $Y$.

We learn $f(\phi(X))$ for each hash function $\phi\in P$ with $Q(q)$ membership queries (when possible) and use the majority rule to find the relevant variables and to recover the target function.∎

See also [49] for a reduction for the randomized non-adaptive learning.

3.4 Reductions from the Exact Learning Model

In this subsection and the next one, we give two other models and show some conditions in which learning in those models can be reduced to learning from membership queries only.

The first learning model is the exact learning model from membership and equivalence queries. In this model, the goal is to learn the target function exactly with membership queries and equivalence queries. In the Equivalence Query (EQ) model, [1], the learning algorithm sends the teacher a hypothesis $h$ from some class of hypothesis $H\supseteq C$. The teacher answers “YES” if $h$ is equivalent to the target $f$, otherwise, it provides the learner a counterexample, i.e., an assignment $a$ where $f(a)\not=h(a)$. We say that a class $C$ is exactly learnable from $H$ from membership and equivalence queries in time $t$, $m$ membership queries and $e$ equivalence queries if there is an algorithm that runs in time $t$, asks at most $m$ membership queries and at most $e$ equivalence queries and outputs $h\in H$ that is equivalent to $f$.

There are several polynomial time exact learning algorithms available in the literature that learns from membership and equivalence queries for classes mentioned in this survey and others. Classes such as Monotone DNF [2], DFA [3], Conjunction of Horn Clauses [17], $O(\log n)$-term DNF [33, 34, 74, 182], read-twice DNF [26], CDNF [33], decision trees [33, 44], Boolean multivariate polynomial [251, 44], multiplicity automata [44], read-once formula [58] and geometric objects [35]. See also references therein and [253]. Some of the algorithms are proper (i.e., $H=C$) and others are non-proper.

The reduction from this model to the model of learning from membership queries only is done as follows.

Lemma 15

Let $C$ and $H\supseteq C$ be two classes of functions. Suppose $C$ is learnable from $H$ from membership and equivalence queries in time $t$, $m$ membership queries and $e$ equivalence queries. Then

If for every $h\in H$ an identity testing set $S_{h}$ for $h$ with respect to $C$ can be constructed in time $t^{\prime}$ and $|S_{h}|\leq s$ then $C$ is learnable in time $poly(t^{\prime},t,m,e,s)$ with $m+es$ membership queries. 2. 2.

If an equivalent test set $S$ for $H$ can be constructed in time $t^{\prime}$ and $|S|=s$ then $C$ is learnable in time $poly(t^{\prime},t,m,e,s)$ with $m+s$ membership queries.

Proof

In 1, the algorithm replaces each equivalence query for $h$ to membership queries to all the assignments in $S_{h}$. If $h$ is consistent with $f$ on $S_{h}$ then $h=f$ and the algorithm outputs $h$. Otherwise, there is $a\in S_{h}$ such that $f(a)\not=h(a)$ and $a$ can be used as an answer to the equivalence query.

In 2, the algorithm first constructs $S$ and asks membership queries to all the assignments in $S$. Then for each equivalence query with $h$, it finds $a\in S$ such that $h(a)\not=f(a)$ and returns the answer $a$ to the algorithm. If no counterexample exists then, it outputs $h$.∎

A hardness result in learning a class $C$ from equivalent and membership queries does not imply hardness in learning $C$ from membership queries only. For example, the hardness result of learning read-thrice DNF, $C$, in [21] cannot be used as a hardness result for this class in the exact learning model from membership queries. This is because, in our definition of efficiency, the query complexity is allowed to be polynomial in ${\rm OPT}_{A}(C)$ which might be exponential in $n$. For example, it is easy to see that ${\rm OPT}_{{\rm AD}}($read-thrice DNF$)=2^{n}$ (even for one term) so this class is optimally learnable (by asking all the queries $\{0,1\}^{n}$). So hardness in learning in this model does not imply hardness in learning from membership queries. On the other hand, hardness results for proper learning, [10], do give hardness results for proper learning from membership queries.

3.5 Reductions from the PAC Learning Model

In this section, we provide a reduction from PAC-learning with membership queries to learning from membership queries only.

In the probably approximately correct learning model (PAC learning model), [268], with membership queries the teacher has a function $f:X\to\{0,1\}$ from some class $C$. The learner can ask the teacher membership queries and is required to learn a function that is $\epsilon$-close (defined below) to the target with high probability. Let $P$ be a distribution on the domain $X$. We say that an algorithm ${\cal A}$ PAC-learns $C$ from $H$ with membership queries according to the distribution $P$ if the algorithm ${\cal A}$, for the input $\epsilon<1$ and $\delta<1$, with probability at least $1-\delta$, outputs a function $h\in H$ that is $\epsilon$-close to $f$, i.e., $\Pr_{P}[f(x)\not=h(x)]\leq\epsilon$.

There are many PAC learning algorithms with membership queries in the literature for classes mentioned in this survey and others. Classes such as decision trees and multivariate polynomials under distributions that support small terms, [68], DNF under the uniform distribution, [130, 172], constant depth circuits under the uniform distribution, [201] and intersections of halfspaces, [189]. If a class is learnable from equivalence and membership queries, then it is PAC-learnable with membership queries according to any distribution, [2].

We say that $H$ is a class of distance $\eta$ with respect to the distribution $P$ if for every $h_{1},h_{2}\in H$, $h_{1}\not=h_{2}$ we have $\Pr_{P}[h_{1}(x)\not=h_{2}(x)]>\eta$. The following lemma shows how to change a PAC-learning algorithm to a learning algorithm with membership queries.

Lemma 16

Let $C$ and $H\supseteq C$ be two classes of functions. Suppose $C$ is PAC-learnable with membership queries from $H$ according to the distribution $P$ in time $t(\epsilon,\delta)$ and $m(\epsilon,\delta)$ membership queries. If $H$ is a class of distance $\eta$ with respect to the distribution $P$, then $C$ is randomized Monte Carlo learnable with $m(\eta,\delta)$ membership queries in time $t(\eta,\delta)$.

Proof

Run the algorithm with $\epsilon=\eta$. Let $h$ be the output. Then with probability at least $1-\delta$ we have $\Pr_{P}[f\not=h]\leq\eta$. Since $C\subseteq H$ we also have $f\in H$ and since $H$ is a class of distance $\eta$ we must have $f=h$.∎

4 Learning $d$-MClause and Group Testing

In this section, we give an example of a class that has been extensively studied in the literature. Consider the class $d$-MClause. That is the class of monotone clauses with at most $d$ variables.

4.1 Group Testing and Applications

In group testing (or pooling design), the task is to determine the positive members $F=\{o_{i_{1}},\ldots,o_{i_{k}}\}$, $k\leq d$, of a set of objects ${\cal O}=\{o_{1},\ldots,o_{n}\}$ by asking queries of the form “does the subset $S\subseteq{\cal O}$ contain a positive object?” That is, “does $F\cap S\not=\emptyset$?”. A negative answer to this question informs the learner that all the items belonging to $S$ are non-positive. The aim of group testing is to identify the unknown subset $F$ using as few queries as possible.

Group testing was originally introduced as a potential approach to economical mass blood testing [111]. However it has been proven to be applicable in a variety of problems, including quality control in product testing [248], searching files in storage systems [190], sequential screening of experimental variables [197], efficient contention resolution algorithms for multiple-access communication [190, 274], data compression [166], and computation in the data stream model [104]. See a brief history and other applications in [85, 113, 114, 217] and references therein.

Group testing is equivalent to learning the class $d$-MClause from membership queries. We have $n=|{\cal O}|$; the target is $\vee_{o_{i}\in F}x_{i}$ and each query “does $F\cap S\not=\emptyset$?” is equivalent to a membership query with the assignment $a\in\{0,1\}^{n}$ where $a_{i}=1$ if and only if $o_{i}\in S$.

4.2 Known Results for Learning $d$-MClause

The following table summarizes the results known for the asymptotic number of membership queries for learning the class $d$-MClause. We will assume that $d\leq\sqrt{n}$.

[TABLE]

There is a (folklore) deterministic adaptive algorithm that runs in polynomial time and asks $O(d\log n)$ queries. See Subsection 4.5. This implies all the results for the upper bounds in the fourth column of the above table. The number of functions in $d$-MClause is

[TABLE]

and therefore by Lemma 2, any Monte Carlo learning algorithm for $d$-MClause must ask at least $\Omega(d\log n)$ queries. This implies all the lower bounds in the table except for the non-adaptive lower bound for the deterministic and Las Vegas algorithm that follows from [120]. See Subsection 4.4. The upper bound of

[TABLE]

for the Monte Carlo non-adaptive algorithm follows from a simple randomized argument. See Subsection 4.4. Porat and Rothschild, [226], gave the first polynomial time deterministic non-adaptive learning algorithm that asks $O(d^{2}\log n)$ queries. The deterministic upper bound for non-adaptive learning follows from a simple probabilistic argument. See Also [123, 124]. The last two results are the deterministic two-round algorithms with $O(d\log n)$ queries and the polynomial time deterministic two-round learning algorithm that asks $\min(d\log^{O(1)}n,d^{1+o(1)}$ $\log n)$ queries. This follows from [56, 81, 170]. See Subsection 4.6.

In particular,

Theorem 4.1

We have

The class $d$-MClause is adaptively optimally learnable. 2. 2.

The class $d$-MClause is non-adaptively almost optimally learnable and optimally learnable in $n$. 3. 3.

The class $d$-MClause is MC randomized adaptively optimally learnable. 4. 4.

The class $d$-MClause is two-round efficiently learnable, two-round optimally learnable in $d$ and two-round optimally learnable in $n$.

4.3 Bounds for OPT($d$-MClause)

Consider a non-adaptive algorithm ${\cal A}$ for learning $d$-MClause and let $x^{(1)},\ldots,x^{(t)}$ be the queries asked by the algorithm. Consider the $t\times n$ matrix $M$ such that its $i$th row is $x^{(i)}$. Let $M_{i}$ be the $i$th column of $M$. If the target function is $x_{i_{1}}\vee\cdots\vee x_{i_{d^{\prime}}}$, $d^{\prime}\leq d$ then the vector of all the answers to the queries is $M_{i_{1}}\vee\cdots\vee M_{i_{d^{\prime}}}$ (bitwise or). Let ${\cal M}$ be the set of all $M_{j_{1}}\vee\cdots\vee M_{j_{d}}$, $j_{i}\in[n]$ for $i=1,\ldots,d$. That is, the set of all possible answers to the queries for all possible target functions. Since each vector in ${\cal M}$ uniquely determines the target function, the matrix $M$ must satisfy the following property: For every $S_{1},S_{2}\subseteq[n]$, where $|S_{1}|,|S_{2}|\leq d$ and $S_{1}\not=S_{2}$, we have

[TABLE]

A matrix that satisfies this property is called $d$-separable matrix. Therefore, ${\rm OPT}_{{\rm NAD}}(d$-Mclause$)$ is equal to the minimum $t$ such that a $d$-separable $t\times n$ matrix exists.

A $t\times n$ matrix $M$ is called $d$-disjunct if for every $d+1$ distinct columns $i_{1},\cdots,i_{d+1}$ there is a row $j$ such that $M_{j,i_{d+1}}=1$ and $M_{j,i_{k}}=0$ for all $k=1,\ldots,d$. It is easy to show that, [190],

[TABLE]

Therefore, it is enough to construct a $d$-disjunct matrix. Using a probabilistic method, [28], it is easy to show that a $d$-disjunct $t\times n$ matrix exists where $t=O(d^{2}\log n)$. Just take each entry in the matrix $M$ to be $1$ with probability $1/d$ and [math] with probability $1-1/d$ and show that for $t=O(d^{2}\log n)$ the probability that the matrix is not $d$-disjunct is less than $1$. This implies

[TABLE]

See also [123, 124, 219]. There is also an almost tight lower bound for $t$ [120, 131, 230]

[TABLE]

4.4 Non-Adaptive Learning $d$-MClause

We now show that the class $d$-MClause is non-adaptively learnable in polynomial time with $O(d^{2}\log n)$ queries. In [226], Porat and Rothschild gave the first polynomial time algorithm for constructing a $d$-disjunct matrix of size $t\times n$ where $t=O(d^{2}\log n)$. Now the learning algorithm is as follows. We first use the Porat and Rothschild [226] algorithm to construct a $d$-disjunct $t\times n$ matrix $M$ of size $t=O(d^{2}\log n)$ in polynomial time. Set $Z=\{x_{1},x_{2},\ldots,x_{n}\}$. Then for every query $M^{(i)}$ (row $i$ in $M$) if $f(M^{(i)})=0$ then for every $j$ where $M^{(i)}_{j}=1$, we remove $x_{j}$ from $Z$. The remaining variables in $Z$ are the variables that appear in the target. This is because if $f=x_{i_{1}}\vee\cdots\vee x_{i_{d^{\prime}}}$ and $i\not\in\{i_{1},\ldots,i_{d^{\prime}}\}$ then the variable $x_{i}$ will be removed from $Z$ by the row $M^{(j)}$ that assign [math] to all $x_{i_{j}}$, $j=1,\ldots,d^{\prime}$ and one to $x_{i}$. Such row exists since $M$ is $d$-disjunct. On the other hand, for every row $M^{(k)}$ where $M^{(k)}_{i_{j}}=1$ for some $j=1,\ldots,d^{\prime}$ we have $f(M^{(k)})=1$ and, therefore, no variable in the target is removed from $Z$. This gives a polynomial time algorithm that asks $O(d^{2}\log n)$ queries. Since the lower bound for the query complexity of non-adaptively learning the class $d$-MClause is $\Omega(d^{2}\log n/\log d)$, the class $d$-MClause is non-adaptively almost optimally learnable and non-adaptively optimally learnable in $n$.

Closing the gap between the lower bound and upper bound is one of the longstanding open problem in group testing. Bshouty proved in [38, 39] that a lower bound of $d^{2}\log n/\omega(1)$ implies that for a power of prime $q=O(d)$ one cannot simulate a black-box multiplication of $d$ elements in the finite field $F_{q^{t}}$ with $O(dt)$ black-box multiplications in $F_{q}$. This is one of the hard problems in algebraic complexity.

For a randomized non-adaptive learning algorithm, just randomly choose $t=O(d(\log n+\log(1/\delta)))$ assignments $a^{(1)},\ldots,a^{(t)}$ in $\{0,1\}^{n}$ where each $a^{(i)}_{j}$ is one with probability $1/d$ and zero with probability $1-1/d$. Define $Z=\{x_{1},\ldots,x_{n}\}$. Then for each assignment that satisfies $f(a^{(i)})=0$ remove all $x_{j}$ from $Z$ for which $a^{(i)}_{j}=1$. It is easy to show that with probability at least $1-\delta$, the variables that remain in $Z$ are the variables of the target.

The problem of strongly attribute learnability of MClause, which is equivalent to the problem of group testing when $d$ is not known to the learner, was studied by Damaschke and Muhammad, [115, 116]. They show that for deterministic non-adaptive algorithms, determining the exact number of the relevant variables $d$ is as difficult as learning the target function. For randomized non-adaptive learning algorithms, they gave the upper bound of $O(\log n)$ to approximate $d$ and the lower bound (with some constraints) of $\Omega(\log n)$.

4.5 Adaptive Learning $d$-MClause

In this subsection, we present the folklore algorithm for adaptively learning the class $d$-MClause. The algorithm runs in polynomial time and has a query complexity that matches the lower bound and therefore $d$-MClause is optimally learnable.

We first give the lower bound

Lemma 17

Any deterministic (or even randomized) algorithm for $d$-MClause must ask at least $d\log(n/d)$ queries.

Proof

Follows from Lemma 2 and the fact that $|C|=\sum_{i=0}^{d}{n\choose i}$.∎

We now give the folklore algorithm. Let $f$ be the target function. For a subset $R\subseteq[n]$, define $1_{R}$ the assignment that is one in the entries that are in $R$ and [math] in the other entries. At the first stage, the algorithm defines a set $S_{1,1}:=[n]$. At stage $i$, the algorithm has disjoint sets $S_{i,1},S_{i,2},\cdots,S_{i,j_{i}}\subseteq[n]$ where $f(1_{S_{i,k}})=1$ for all $k=1,\ldots,j_{i}$. The algorithm at stage $i$ partitions each set $S_{i,k}$, $k=1,\ldots,j_{i}$ into two (almost) equal disjoint sets $S_{i,k,1}\cup S_{i,k,2}$ and asks two queries $f(1_{S_{i,k,1}})$ and $f(1_{S_{i,k,2}})$. The sets that will survive to the following stage, $i+1$, are the sets $T$ for which $f(1_{T})=1$. Those will be assigned to $S_{i+1,1},\ldots,S_{i+1,j_{i+1}}$. The algorithm stops when the sizes of those sets are $1$. Then each $S_{i+1,k}$ will be holding an index of a variable in the target.

Obviously, throughout the algorithm we have $j_{i}\leq d$ and $|S_{i,k}|\leq\lceil n/2^{i-1}\rceil$ for all $k=1,2,\ldots,j_{i}$. The algorithm has at most $\log n$ stages. At each stage, it asks at most $2d$ queries, and therefore, the total number of queries is

[TABLE]

A more precise analysis gives the upper bound $2d\log(n/d)+O(d).$

See also the algorithms in [61, 90, 119, 125, 254, 266] and references therein.

The above adaptive algorithm runs even if $d$ is unknown to the learner. Therefore, the class $d$-MClause is adaptively strongly attribute-optimally learnable. This implies

Theorem 4.2

[160]** The class $d$-MClause is strongly attribute-optimally learnable with $O(d\log(n/d))$ queries.

For randomized adaptive algorithms see [117] and reference within. When $d$ is unknown, Cheng [82] shows that there is a randomized adaptive learning algorithm that asks $O(d\log d)$ queries and finds $d$ with probability at least $1-1/d^{\Theta(1)}$.

4.6 Two-Round Learning

In [56], De Bonis et al. shows that there is a two-round adaptive algorithm for learning $d$-MClause that asks $O(d\log n)$ queries. See also [88, 126]. This is asymptotically as efficient as the best fully adaptive learning algorithms. Therefore

[TABLE]

The algorithm uses $(2d,d+1,n)$-selector. A $(d,m,n)$-selector is a $t\times n$ Boolean matrix such that any $d$ columns contain at least $m$ distinct rows of Hamming weight $1$. It is known that there is a $(2d,d+1,n)$-selector of size $t=O(d\log n)$. This follows from the following simple probabilistic argument: randomly choose $t\times n$ matrix where each entry is $1$ with probability $p=1/(2d)$ and [math] with probability $1-p$. Then show that the probability that the matrix is not a $(2d,d+1,n)$-selector is less than one.

Given a $(2d,d+1,n)$-selector, the algorithm is as follows. Let $f=x_{i_{1}}\vee\cdots\vee x_{i_{d^{\prime}}}$, $d^{\prime}\leq d$ be the target function. At the first round, the algorithm asks $t=O(d\log n)$ queries that are the rows of a $(2d,d+1,n)$-selector. Let $X=\{x_{1},\ldots,x_{n}\}$. The algorithm then eliminates all the variables $x_{i}$ in $X$ where there is a query $a$ for which $f(a)=0$ and $a_{i}=1$. At the second round, for each variable $x_{i}\in X$ (that was not eliminated in the first round) the algorithm asks the query $e^{(i)}$ where $e^{(i)}_{i}=1$ and $e^{(i)}_{j}=0$ for all $j\not=i$. Then $x_{i}$ in the target if and only if $f(e^{(i)})=1$.

Now we show that the number of variables that are not eliminated in the first round is at most $2d-1$. Suppose for the contrary that there are $2d$ variables $X^{\prime}=\{x_{j_{1}},\ldots,x_{j_{2d}}\}$ that are not eliminated in the first round. By the same argument as in Subsection 4.4, $\{x_{i_{1}},\ldots,x_{i_{d^{\prime}}}\}\subset X^{\prime}$. By the property of $(2d,d+1,n)$-selectors and since $d^{\prime}\leq d$ there is an assignment $a$ where $a_{i_{1}}=\cdots=a_{i_{d^{\prime}}}=0$ and $a_{j_{k}}=1$ for some $j_{k}\not\in\{i_{1},\ldots,i_{d^{\prime}}\}$. This implies that $f(a)=0$ and $a_{j_{k}}=1$ and the variable $x_{j_{k}}$ was eliminated in the first round. This is a contradiction.

Indyk shows in [170] how to construct an explicit $(2d,d+1,n)$-selector of size $d\cdot poly(\log n)$. This construction gives a polynomial time learning algorithm for $d$-MClause with $d\cdot poly(\log n)$ queries. Therefore the class $d$-MClause is two-round efficiently learnable and two-round optimally learnable in $d$. Cheraghchi, [81], used recent results in extractors to prove that $d$-MClause is two-round almost optimally learnable in $n$. His algorithm asks $d^{1+o(1)}\log n$ queries.

4.7 Other Related Problems

The group testing with inhibitors (GTI) model was introduced in [133]. In this model, in addition to positive items and regular items, there is also a category of items called inhibitors. The inhibitors are the items that interfere with the test by hiding the presence of positive items. As a consequence, a test yields a positive feedback if and only if the tested pool contains one or more positives and no inhibitors. This problem is equivalent to learning functions of the form

[TABLE]

This problem is studied in [76, 94, 112, 133, 162, 167].

See other related problems in [86, 217] and references therein.

5 Learning $s$-Term $r$-Monotone DNF

Consider the class $s$-term $r$-MDNF. That is, the class of monotone DNF with $s$ monotone terms (monomials) where each term is of size at most $r$. Torney, [262], first introduced the problem and gave some applications in molecular biology. In this section, we present some results known from the literature for learning this class.

5.1 Learning a Hypergraph and its Applications

A hypergraph is $H=(V,E)$ where $V$ is the set of vertices, and $E\subseteq 2^{V}$ is the set of edges. The dimension or rank of the hypergraph $H$ is the cardinality of the largest set in $E$. A hypergraph is called Sperner hypergraph if no edge is a subset of another. For a set $S\subseteq V$, the edge-detecting queries $Q_{H}(S)$ is answered “Yes” or “No”, indicating whether $S$ contains all the vertices of at least one edge of $H$. Learning the class $s$-term $r$-MDNF is equivalent to learning a Sperner hidden hypergraph of dimension at most $r$ with at most $s$ edges using edge-detecting queries [14].

This problem has many applications in chemical reactions, molecular biology, and genome sequencing. In chemical reactions, we are given a set of chemicals, some of which react and some which do not. When multiple chemicals are combined in one test tube, a reaction is detectable if and only if at least one set of the chemicals in the tube reacts. The goal is to identify which sets react using as few experiments as possible.

See [6, 11, 12, 14, 15, 42, 55, 87, 91, 94, 100, 114, 115, 122, 139, 144, 211, 212, 235, 262] for more details on the problem, learnability of subclasses of $s$-term $r$-MDNF and other applications. This problem is also called, “sets of positive subsets” [262] “complex group testing” [114, 211] and “group testing in hypergraph” [139].

In all of the above applications, the size of the terms $r$ is much smaller than the number of terms $s$ and both are much smaller than the number of vertices $n$. Therefore, all the results in the literature, except [13], assumes that $r,s\leq\sqrt{n}$, although they do not mention this constraint explicitly. For ease of the presentation of the results, we will also adopt this constraint throughout this section.

5.2 Cover Free Families

One of the tools used in the literature for learning $s$-term $r$-MDNF is cover-free families (CFF). A $(n,(s,r))$-cover free family ($(n,(s,r))$-CFF), [190], is a set $A\subseteq\{0,1\}^{n}$ such that for every $1\leq i_{1}<i_{2}<\cdots<i_{d}\leq n$ where $d=s+r$ and every $J\subseteq[d]$ of size $|J|=s$ there is $a\in A$ such that $a_{i_{k}}=0$ for all $k\in J$ and $a_{i_{j}}=1$ for all $j\not\in J$. Denote by $N(n,(s,r))$ the minimum size of such set. The lower bound in [123, 215, 260] is

[TABLE]

where

[TABLE]

It is known, [12], that a set of

[TABLE]

random vectors $a\in\{0,1\}^{n}$, where each $a_{j}$ is $1$ with probability $r/(s+r)$, is a $(n,(s,r))$-CFF with probability at least $1-\delta$.

It follows from [38, 41, 52, 134] that there is a polynomial time (in the size of the CFF) deterministic construction of $(n,(s,r))$-CFF of size

[TABLE]

where the $o(1)$ is with respect to $\min(r,s)$. When $r=o(s)$, the construction can be done in linear time [41, 52].

5.3 Non-Adaptive Learning $s$-Term $r$-MDNF

In this section, we give a non-adaptive learning algorithm for the class of $s$-term $r$-MDNF.

We first give a lower bound

Theorem 5.1

[13, 114]** Let $n\geq r+s$. Any equivalent test set $A\subseteq\{0,1\}^{n}$ for $s$-term $r$-MDNF is $(n,(s,r-1))$-CFF and $(n,(s-1,r))$-CFF. Therefore, Any non-adaptive algorithm for learning $s$-term $r$-MDNF must ask at least

[TABLE]

queries.

In particular, when $r$ is constant, the number of queries is at least

[TABLE]

Proof

Consider any distinct $1\leq i_{1},\cdots,i_{r+s-1}\leq n$. To be able to distinguish between the two functions $f_{1}=(x_{i_{1}}\cdots x_{i_{r}})\vee x_{i_{r+1}}\vee\cdots\vee x_{i_{r+s-1}}$ and $f_{2}=(x_{i_{1}}\cdots x_{i_{r-1}})\vee x_{i_{r+1}}\vee\cdots\vee x_{i_{r+s-1}}$ we must have an assignment $a$ that satisfies $a_{i_{1}}=\cdots=a_{i_{r-1}}=1$ and $a_{i_{r}}=\cdots=a_{i_{r+s-1}}=0$. Therefore $A$ is $(n,(s,r-1))$-CFF.

To be able to distinguish between the two functions $g_{1}=(x_{i_{1}}\cdots x_{i_{r}})\vee x_{i_{r+1}}\vee\cdots\vee x_{i_{r+s-1}}$ and $g_{2}=x_{i_{r+1}}\vee\cdots\vee x_{i_{r+s-1}}$ we must have an assignment $a$ that satisfies $a_{i_{1}}=\cdots=a_{i_{r}}=1$ and $a_{i_{r+1}}=\cdots=a_{i_{r+s-1}}=0$. Therefore $A$ is $(n,(s-1,r))$-CFF.∎

We now give a simple upper bound

Theorem 5.2

Any $(n,(s,r))$-CFF $A$ is an equivalent test set for $s$-term $r$-MDNF. Therefore, there is a non-adaptive learning algorithm for $s$-term $r$-MDNF that asks

[TABLE]

queries. In particular, when $r$ is constant,

[TABLE]

Proof

Let $A$ be a $(n,(s,r))$-CFF. Let $f_{1},f_{2}$ be any two non-equivalent $s$-term $r$-MDNF. Suppose $f_{1}=M_{1,1}\vee M_{1,2}\vee\cdots\vee M_{1,s_{1}}$ and $f_{2}=M_{2,1}\vee M_{2,2}\vee\cdots\vee M_{2,s_{2}}$ where $s_{1},s_{2}\leq s$. Let $b\in\{0,1\}^{n}$ be an assignment such that (w.l.o.g.) $f_{1}(b)=1$ and $f_{2}(b)=0$. Then $M_{2,i}(b)=0$ for all $i=1,\ldots,s_{2}$ and $M_{1,j}(b)=1$ for some $1\leq j\leq s_{1}$. Let $M_{1,j}=x_{k_{1}}x_{k_{2}}\cdots x_{k_{r^{\prime}}}$, $r^{\prime}\leq r$. Then for every $M_{2,i}$, $i=1,\ldots,s_{2}$, there is a variable $x_{j_{i}}$ in $M_{2,i}$ where $b_{j_{i}}=0$ and for all the variables $x_{k_{1}},x_{k_{2}},\ldots,x_{k_{r^{\prime}}}$ in $M_{1,j}$ we have $b_{k_{1}}=\cdots=b_{k_{r^{\prime}}}=1$.

Now take $a\in A$ such that $a_{j_{i}}=0$, $i=1,\ldots,s_{2}$ and $a_{k_{1}}=a_{k_{2}}=\cdots=a_{k_{r^{\prime}}}=1$. Such $a$ exists since $A$ is $(n,(s,r))$-CFF. Then we have $f_{1}(a)=1$ and $f_{2}(a)=0$. This completes the proof.∎

The first explicit non-adaptive learning algorithm for $s$-term $r$-MDNF was given by Gao et al., [139]. They show that this class can be learned with $(n,(s,r))$-CFF. Given such a $(n,(s,r))$-CFF, the algorithm simply takes all the monomials $M$ of size at most $r$ that satisfy $(\forall a\in A)(M(a)=1\Rightarrow f(a)=1)$. It is easy to see that the disjunction of all such monomials is equivalent to the target function. Assuming a set of $(n,(s,r))$-CFF of size $N$ can be constructed in time $T$, the above algorithm learns $s$-term $r$-MDNF with $N$ queries in time $O({n\choose r}N+T)$. This with (12) gives

Theorem 5.3

There is a non-adaptive learning algorithm for $s$-term $r$-MDNF that asks

[TABLE]

queries and runs in $poly({n\choose r},Q)$ time.

When $r$ is constant, the algorithm asks

[TABLE]

queries and runs in $poly(n)$ time.

In particular, for constant $r$, the class $s$-term $r$-MDNF is non-adaptively almost optimally learnable and optimally learnable in $n$.

When $r=\omega(1)$ we can use Lemma 14 to prove

Theorem 5.4

Let $r=\omega(1)$. There is a non-adaptive learning algorithm for $s$-term $r$-MDNF that asks

[TABLE]

queries and runs in $poly(n,{2(rs)^{2}\choose r})$ time.

In particular, for $r\leq s^{c}$ for some $c<1$, the class $s$-term $r$-MDNF is non-adaptively almost optimally learnable and optimally learnable in $n$.

Proof

Follows from Lemma 14 and Theorem 5.3 and the fact that any $s$-term $r$-MDNF has at most $sr$ relevant variables.∎

One can now use (11) in a straightforward manner to get a randomized non-adaptive algorithm with better time and query complexity. Recently, Abasi et al. [13], gave an almost optimal learning algorithm for all $n,r$ and $s$.

5.4 Adaptive Learning $s$-Term $r$-MDNF

In this section, we give results on adaptive algorithms for learning $s$-term $r$-MDNF.

Adaptive algorithms for learning $s$-term $r$-MDNF is studied in [14, 15] and [12]. The information theoretic lower bound for this class is $rs\log n$. Angluin and Chen gave in [15] the lower bound $\Omega((2s/r)^{r/2}+rs\log n)$ when $s>r$ and Abasi et al. gave in [12] the lower bound $\Omega(({r}/{s})^{s-1}+rs\log n)$ when $s\leq r$. Angluin and Chen gave a polynomial time adaptive algorithm for learning $s$-term $2$-MDNF that asks $O(s\log n)$ queries. Therefore, the class $s$-term $2$-MDNF is adaptively optimally learnable. In [12] Abasi et al. gave a polynomial time learning algorithm for $s$-term $r$-MDNF that asks $rs\log n+({r}/{s})^{s+o(s)}$ queries when $r>s$ and $rs\log n+s^{r/2+o(r)}$ queries when $r\leq s$. They also gave some randomized algorithms.

The following table summarizes the latest results: Det. and Rand. stand for deterministic algorithm and randomized algorithm, respectively.

[TABLE]

5.5 Learning Subclasses of $s$-term $r$-MDNF

Learning subclasses of graphs and hypergraphs from edge-detecting queries received considerable attention in the literature due to its diverse applications [14, 15, 42, 87]. This is equivalent to learning subclasses of $s$-term $2$-MDNF and $s$-term $r$-MDNF, respectively. Subclasses include Graphs of bounded degree [15], Hamiltonian cycles [87, 143, 144], matchings [11, 42, 87], stars [6, 87], cliques [6, 87], families of graphs that are closed under isomorphism [11] $r$-uniform hypergraph and almost uniform hypergraph [15]. The class of Read-Once $2$-MDNF is equivalent to learning matchings [11].

6 Learning Decision Tree

In this section we study the learnability of the class of Depth $d$-DT (${\rm DT}_{d}$), i.e., the class of all decision trees of depth at most $d$.

6.1 Bounds on ${\rm OPT}({\rm DT}_{d})$

We say that a set of assignments $A\subseteq\{0,1\}^{n}$ is $(n,d)$-universal set if for every $1\leq i_{1}<i_{2}<\cdots<i_{d}\leq n$ and every $b\in\{0,1\}^{d}$ there is an $a\in A$ such that $a_{i_{j}}=b_{j}$ for all $j=1,\ldots,d$.

It is known that any $(n,d)$-universal set is of size $\Omega(2^{d}\log n)$ [191, 246]. The probabilistic method with the union bound gives the upper bound $O(d2^{d}\log n)$. The best known polynomial time, $poly(n,2^{d})$, construction gives an $(n,d)$-universal set of size $O(2^{d+\log^{2}d}\log n)$ [218]. It is easy to show that a random uniform set $A\subseteq\{0,1\}^{n}$ of size $O(d2^{d}\log n)$ is $(n,d)$-universal set with probability at least $1/n^{O(d)}$. For $d\leq\log n/\log\log n$, an $(n,d)$-universal set of size $d^{O(1)}2^{d}\log n$ can be constructed in polynomial time [218].

We now prove

Theorem 6.1

A set of assignments $A\subseteq\{0,1\}^{n}$ is a hitting set for ${\rm DT}_{d}$ if and only if $A$ is an $(n,d)$-universal set. In particular

[TABLE]

Proof

Let $A$ be an $(n,d)$-universal set. Let $f\not\equiv 0$ be a decision tree of depth at most $d$. Consider a path $p$ from the root of $f$ to a leaf that is labeled with $1$. Let $p=v_{1}\stackrel{{\scriptstyle{\scriptscriptstyle{\xi_{1}}}}}{{\to}}v_{2}\stackrel{{\scriptstyle{\scriptscriptstyle{\xi_{2}}}}}{{\to}}\cdots\stackrel{{\scriptstyle{\scriptscriptstyle{\xi_{d^{\prime}-1}}}}}{{\to}}v_{d^{\prime}}\stackrel{{\scriptstyle{\scriptscriptstyle{\xi_{d^{\prime}}}}}}{{\to}}v_{d^{\prime}+1}$, $d^{\prime}\leq d$ where $v_{d^{\prime}+1}$ is the leaf that is labeled with $1$, $v_{i}$ is labeled with $x_{j_{i}}$ and the edge $v_{j}\to^{\xi_{j}}v_{j+1}$ is labeled with $\xi_{j}\in\{0,1\}$. If $A$ is $(n,d)$-universal set then there is $a\in A$ such that $a_{j_{i}}=\xi_{i}$ for all $i=1,\ldots,d^{\prime}$ and $f(a)=1$. Therefore, $A$ is a hitting set for $DT_{d}$.

The other direction follows from the fact that any term $T=x_{j_{1}}^{\xi_{1}}\cdots x_{j_{d}}^{\xi_{d}}$ where $1\leq j_{1}<\cdots<j_{d}\leq n$ and $\xi_{i}\in\{0,1\}$ is a decision tree of depth $d$. Recall that $x^{\xi}=x$ if $\xi=1$ and $x^{\xi}=\bar{x}$ if $\xi=0$. To hit $T$ we need an assignment $a$ that satisfies $a_{j_{1}}=\xi_{1},\ldots,a_{j_{d}}=\xi_{d}$.∎

For equivalent test set, adaptive and non-adaptive learning, we prove

Theorem 6.2

We have

[TABLE]

Proof

Let ${\cal A}$ be an adaptive algorithm that learns DTd. We run ${\cal A}$ and answer [math] for all the membership queries. The algorithm must stop and return the function [math] as the target function. Let $A$ be the set of all the assignments asked in the membership queries. If $A$ is not $(n,d)$-universal set, then there is $1\leq j_{1}<\cdots<j_{d}\leq n$ and $(\xi_{1},\ldots,\xi_{d})\in\{0,1\}^{d}$ such that $(a_{j_{1}},\ldots,a_{j_{d}})\not=(\xi_{1},\ldots,\xi_{d})$. Then the term $T=x_{j_{1}}^{\xi_{1}}\cdots x_{j_{d}}^{\xi_{d}}\in{\rm DT}_{d}$ is zero on all the assignments of $A$, and we get a contradiction.

We now prove that ${\rm DT}_{d}\oplus{\rm DT}_{d}\subset{\rm DT}_{2d}$ and then the other results follow from 3 in Lemma 5.

For any two functions $f,g\in{\rm DT}_{d}$ and the corresponding decision trees $T(f)$ and $T(g)$, one can construct a tree for $f\oplus g$ from $T(f)$ and $T(g)$ as follows. First $g+1$ is equivalent to the tree $T(g+1)$ that is the same as $T(g)$ where the labels in the leafs are flipped from [math] to $1$ and from $1$ to [math]. Now in the tree $T(f)$, replace each leaf that is labeled with [math] with the tree $T(g)$ and each leaf that is labeled with $1$ with the tree $T(g+1)$. It is easy to show that this tree computes $f\oplus g$, and its depth is at most $2d$.∎

6.2 Adaptive Learning Decision Tree

The adaptive learnability of decision tree of depth $d$ follows from many papers [38, 39, 65, 68, 130, 172, 187, 251]. One of the powerful techniques used in the literature is the discrete Fourier transform DFT. In DFT, one regards the Boolean function $f:\{0,1\}^{n}\to\{0,1\}$ as a real function $f:\{0,1\}^{n}\to\{+1,-1\}$ in $\Re^{\{0,1\}^{n}}$, represent it as a linear combination of orthonormal basis functions, and then learns the coefficients.

In [187], Kushilevitz and Mansour used this technique for learning the class of decision trees as follows. Consider the set ${\cal A}=\{\chi_{a}:\{0,1\}^{n}\to\{-1,+1\}|a\in\{0,1\}^{n}\}$ where $\chi_{a}(x)=\prod_{i=1}^{n}(-1)^{a_{i}x_{i}}$. It is easy to see that ${\cal A}$ is an orthonormal basis for the set of functions $f:\{0,1\}^{n}\to\Re$. Therefore, every function $f:\{0,1\}^{n}\to\Re$ can be represented as

[TABLE]

where $\hat{f}(a)\in\Re$. This representation is called the Fourier representation of $f$ and $\hat{f}(a)$ is called the Fourier coefficient of $\chi_{a}$. It is easy to see that the Fourier coefficients of $\chi_{a}$ is $\hat{f}(a)={\bf E}_{x}[f(x)\chi_{a}(x)]$ where $x\in\{0,1\}^{n}$, and the expectation is over the uniform distribution on $\{0,1\}^{n}$. So every coefficient can be estimated using Chernoff bound. It remains to show that for a decision tree of depth $d$ the number of nonzero Fourier coefficients $\hat{f}(a)$ is small, and they can be found exactly and efficiently222An efficient learning algorithm for decision tree of depth $d$ is one that asks $poly(2^{d},\log n)$ queries and runs in $poly(2^{d},n)$ time. See Section 1.5.. We demonstrate the algorithm with the help of the following simple example.

[TABLE]

[TABLE]

[TABLE]

Consider the decision tree in Figure 4: $f=x_{4}(x_{2}+\bar{x}_{2}x_{3})+\bar{x}_{4}(x_{3}+\bar{x}_{3}\bar{x}_{1})$. In this example, the depth of $f$ is $3$, and $f$ is a sum of terms of size $3$. First notice that since the terms are disjoint (no two terms are equal to $1$ for the same assignment), the “+” operation can be replaced by the arithmetic “+” operation in $\Re$. To change the values of the function to $+1,-1$ values, we take $g=1-2f$. In general, every decision tree of size $s$ and depth $d$ can be written as a sum (in $\Re$) of $s$ terms of size at most $d$. Now take any term, say $t=x_{4}\bar{x}_{2}x_{3}$. Over the real numbers $\Re$, we can express $x$ as $(1-(-1)^{x})/2$ and $\bar{x}$ as $(1+(-1)^{x})/2$. Then the term $t$ can be expressed as

[TABLE]

In general, every term of size $d$ has a Fourier representation that contains $2^{d}$ non-zero coefficients $\hat{f}(a)$ each is $\pm 1/2^{d}$ and $wt(a)\leq d$ where $wt(a)$ is the Hamming weight of $a$, i.e, the number of ones in $a$. Therefore, every decision tree of size $s$ and depth $d$ has a Fourier representation that contains at most $s2^{d}+1\leq 2^{2d}+1$ non-zero Fourier coefficients, each has one of the values in $\{\pm k/2^{d}\ |\ k\in[s2^{d}]\}\cup\{0\}$. In fact, using Parseval’s identity, one can prove that $\sum_{a\in\{0,1\}^{n}}\hat{f}(a)^{2}=1$ and, therefore, the Fourier coefficients have values from $\{\pm k/2^{d}\ |\ k\in[2^{d}]\}\cup\{0\}$. Also each non-zero coefficient $\hat{f}(a)$ satisfies $wt(a)\leq d$. Now using Chernoff bound, for each assignment $a$ of weight at most $d$, one can exactly find each coefficient $\hat{f}(a)={\bf E}_{x}[f(x)\chi_{a}(x)]$ with $poly(s,2^{d})$ queries. The problem with this algorithm is that since the number of assignments $a$ that satisfies $wt(a)\leq d$ is $n^{O(d)}$ the time complexity is exponential $n^{O(d)}$.

Kushilevitz and Mansour in [187] and Goldreich and Levin in [149] gave an adaptive algorithm that finds the non-zero coefficients in $poly(2^{d},n)$ time and queries. Kushilevitz and Mansour algorithm (KM-algorithm) is based on the fact that for any $\alpha\in\{0,1\}^{r}$ we have

[TABLE]

where for $y=(y_{1},\ldots,y_{r})$ and $x=(x_{1},\ldots,x_{s})$, $yx=(y_{1},\ldots,y_{r},x_{1},\ldots,x_{s})$. Notice that $F_{\alpha}\leq 1$ can be computed exactly (with high probability) with Chernoff bound. Now KM-algorithm uses divide and conquer technique with the above identity to find the non-zero coefficients in $poly(2^{d},n)$ time. Notice that

[TABLE]

and therefore $F_{()}=F_{(0)}+F_{(1)}$. The algorithm first computes $F_{(0)}$ and $F_{(1)}$ and lets $T_{1}=\{\xi\in\{0,1\}|F_{(\xi)}\not=0\}$. At some stage, it holds a set $T_{r}=\{\alpha^{(1)},\ldots,\alpha^{(t)}\}\subseteq\{0,1\}^{r}$ where $F_{\alpha}\not=0$ for all $\alpha\in T_{r}$ and $F_{\alpha}=0$ for all $\alpha\not\in T_{r}$. Now for each $\alpha\in T_{r}$ it computes $F_{(\alpha,0)}$ and $F_{(\alpha,1)}$. Since $0\not=F_{\alpha}=F_{(\alpha,0)}+F_{(\alpha,1)}$, at least one of them is not zero. Then it defines $T_{r+1}=\{(\alpha,\xi)\ |\ F_{(\alpha,\xi)}\not=0\}$. Since the number of the non-zero Fourier coefficients of a decision tree of depth at most $d$ is less than $2^{2d}+1$, the number of elements in $T_{r+1}$ is less than $2^{2d}+1$. Notice that for $\alpha\in\{0,1\}^{n}$

[TABLE]

and therefore, $T_{n}$ contains all the assignments $\alpha\in\{0,1\}^{n}$ for which $\hat{f}(\alpha)\not=0$. It is easy to see that this algorithm runs in time $poly(2^{d},n)$. Therefore

Theorem 6.3

[187]** There is an adaptive Monte Carlo learning algorithm that learns ${\rm DT}_{d}$ in $poly(2^{d},n)$ time and $O(2^{6d}\cdot n)$ membership queries.

Kushilevitz and Mansour use a derandomization technique to change the algorithm to deterministic. They prove

Theorem 6.4

[187]** There is an adaptive deterministic learning algorithm that learns ${\rm DT}_{d}$ in $poly(2^{d},n)$ time and $O(2^{10d}\cdot n\log^{2}n)$ membership queries.

By Lemma 9, we have

Theorem 6.5

There is an adaptive deterministic learning algorithm that learns ${\rm DT}_{d}$ in $poly(2^{d},n)$ time and $O(2^{13d+o(d)}\cdot\log n)$ membership queries.

In particular, ${\rm DT}_{d}$ is efficiently adaptively learnable and optimally adaptively learnable in $n$.

Proof

We use Lemma 9. Since decision trees of depth $d$ have at most $2^{d}$ relevant variables, we can set $r=2^{d}$. By Theorem 6.4, $Q(r)=O(d^{2}2^{11d})$. An $(n,d)$-universal set is a constant testing set for ${\rm DT}_{d}$. See the proof of Theorem 6.2. The best known polynomial time, $poly(n,2^{d})$, construction gives an $(n,d)$-universal set of size $O(2^{d+o(d)}\log n)$ [218]. Therefore $P(n,r)=O(2^{d+o(d)}\log n)$. Then the reduction in Lemma 9 gives a polynomial time adaptive learning algorithm that asks

[TABLE]

membership queries.∎

See other randomized algorithms in [68, 251] that use different techniques. The algorithm in [251] uses membership and equivalence queries, and it is easy to see that every equivalence query can be simulated by randomized membership queries.

6.3 Non-Adaptive Learning Decision Tree

In this subsection, we give a sketch of the results in [65, 149] and then of [130] that gave the first polynomial time Monte Carlo non-adaptive learning algorithm for ${\rm DT}_{d}$.

The following is the result of Hofmeister in [159]

Lemma 18

[159]** There is a polynomial time deterministic non-adaptive algorithm for $C=\{\chi_{a}(x)|wt(a)\leq d\}$ that asks $O(d\log n)$ membership queries.

In particular, there is a set of assignments $B=\{b^{(1)},\ldots,b^{(t)}\}$ of size $t=O(d\log n)$ that can be constructed in polynomial time and an algorithm ${\cal A}$ such that: Given $\chi_{a}(b^{(1)}),\ldots,\chi_{a}(b^{(t)})$ for some $a$ of weight at most $d$, the algorithm ${\cal A}$ finds the assignment $a$ in polynomial time.

The main idea of the learning algorithm of ${\rm DT}_{d}$ is to use pairwise independent assignments for estimating ${\bf E}_{x}[f(x)\chi_{a}(x)]$, rather than totally independent assignments. Since pairwise independent assignments can be generated with a small number of random bits, the problem is reduced to finding the Fourier coefficient of a function that depends on a small number of variable. Using those coefficients one can recover the assignments $a$ with large Fourier coefficients $\hat{f}(a)$ in $f$. We now give a sketch of the algorithm and its correctness.

It is easy to see that for the function $f_{b}(x):=f(x+b)$ we have

[TABLE]

Therefore

[TABLE]

where ${\rm sign}(x)=1$ if $x\geq 0$ and $=-1$ otherwise. Therefore

[TABLE]

Assuming we know ${\rm sign}(\hat{f}(a))$, to compute $\chi_{a}(b)$ for some $b$ we only need to know the sign of ${\bf E}_{x}[f_{b}(x)\chi_{a}(x)]$. Notice that $|{\bf E}_{x}[f_{b}(x)\chi_{a}(x)]|=|\hat{f}_{b}(a)|=|\hat{f}(a)|$. Now assuming $|{\bf E}_{x}[f_{b}(x)\chi_{a}(x)]|=|\hat{f}(a)|$ is not zero (and therefore $\geq 1/2^{d}$), to compute the sign of ${\bf E}_{x}[f_{b}(x)\chi_{a}(x)]$ it is enough to use Bienayme-Chebyshev bound rather than Chernoff bound. That is, to estimate $f_{b}(x)\chi_{a}(x)$ on pairwise independent assignments rather than totally independent assignments. To generate such assignments, consider a random uniform $k\times n$ matrix $R$ over the binary field $F_{2}=\{0,1\}$ where $k=\log m$. We will determine $m$ later. Then the assignments in the set $\{pR\ |\ p\in\{0,1\}^{k}\}$ are pairwise independent. Combining all the above ideas with Bienayme-Chebyshev bound we prove

Lemma 19

With probability at least $1-4/(m(\hat{f}(a))^{2})$ we have

[TABLE]

where $f_{R,b}(x_{1},\ldots,x_{k})=f(xR^{T}+b)$.

Proof

Let $\{z_{1},\ldots,z_{m}\}=\{pR|p\in\{0,1\}^{k}\}$. Consider a random uniform $x\in\{0,1\}^{n}$ and the random variable $X(x)=f(x)\chi_{a}(x)$. We have $\mu={\bf E}[X]=\hat{f}(a)$ and $\sigma={\bf Var}[X]={\bf E}[X^{2}]-{\bf E}[X]^{2}=1-\hat{f}(a)^{2}$. Since $X_{i}=X(z_{i})$ are pairwise independent, by Bienayme-Chebyshev bound we get

[TABLE]

Therefore, with probability at least $1-2/(m\hat{f}(a)^{2})$ we have

[TABLE]

By (13), with probability at least $1-4/(m\hat{f}(a)^{2})$ we have

[TABLE]

Since

[TABLE]

The result follows.∎

Notice that $f_{R,b}$ is a function in $k=\log m$ variables. This is the key lemma. It shows that if $\hat{f}(a)\not=0$, there is a positive probability that $\chi_{a}(b)$ can be computed (modulo the ${\rm sign}(\hat{f}(a))$) using the sign of some Fourier coefficient of $\hat{f}_{R,b}$. Since $f_{R,b}$ depends on a small number of variable and a membership to $f_{R,b}$ can be simulated by a membership to $f$ (since $f_{R,b}(a_{1},\ldots,a_{k})=f((a_{1},\ldots,a_{k})R^{T}+b)$), all its Fourier coefficients can be easily found. Now if $\chi_{a}(b)$ is computed for all $b\in B=\{b^{(1)},\ldots,b^{(t)}\}$, where $B$ is the set in Lemma 18, then $a$ can be found with Hofmeister’s algorithm. In the following, we give more details.

By Lemma 19 and using the union bound we have

Lemma 20

Let $B=\{b^{(1)},\ldots,b^{(t)}\}$ be as in Lemma 18 and $t=|B|$. Let $m=2^{2d+3}t$. For any $a$, if $|\hat{f}(a)|\geq 1/2^{d}$, there is $z\in\{0,1\}^{k}$ and $\xi\in\{-1,+1\}$ such that with probability at least $1-(2^{2d+2}t)/m\geq 1/2$ we have

[TABLE]

where $f_{R,b^{(i)}}(x_{1},\ldots,x_{k})=f(xR^{T}+b^{(i)})$.

Now since $f_{R,b^{(i)}}$ depends on $k=\log m$ variables we can find all its Fourier coefficients in time $poly(m)$ and $2^{k}=m=O(2^{2d}d\log n)$ membership queries. Therefore, in $poly(m,t)$ time and $2^{k}t=O(2^{2d}d^{2}\log^{2}n)$ membership queries we can find

[TABLE]

for all $z$ and $\xi$. If $\hat{f}(a)$ is not zero then $|\hat{f}(a)|\geq 1/2^{d}$ and then by Lemma 20, with probability at least $1/2$, some $z$ and $\xi$ satisfies $(\chi_{a}(b^{(1)}),\ldots,\chi_{a}(b^{(t)}))=W_{\xi,z}$. Then by Lemma 18, $a$ can be recovered in polynomial time. So from all $W_{\xi,z}$ and using the algorithm ${\cal A}$ in Lemma 18 we find a set of assignments $Q$ such that: if $\hat{f}(a)$ is not zero then $a\in Q$ with probability at least $1/2$. This implies that on average, $Q$ contains half of the assignments that correspond to the non-zero Fourier coefficients of $f$. The size of $Q$ is at most $2^{k+1}=O(2^{2d}d\log n)$. Then we find the Fourier coefficient $\hat{f}_{a}={\bf E}_{x}[f(x)\chi_{a}(x)]$ for all $a\in Q$ using Chernoff bound and the union bound with $O(2^{d}\log|Q|)=o(2^{k})$ additional membership queries. We can repeat the above $\log(2^{2d}/\delta)$ time to find all the non-zero Fourier coefficients of $f$ with probability at least $1-\delta$.

Putting all the above ideas together, it follows that

Lemma 21

[130]**. There is a non-adaptive Monte Carlo learning algorithm that learns ${\rm DT}_{d}$ in polynomial time and $O(d^{3}2^{2d}\log^{2}n)$ membership queries.

By Lemma 14, we get

Lemma 22

There is a non-adaptive Monte Carlo learning algorithm that learns ${\rm DT}_{d}$ in polynomial time and $O(d^{4}2^{4d}\log n)$ membership queries.

In particular, ${\rm DT}_{d}$ is MC efficiently non-adaptively learnable and MC optimally non-adaptively learnable in $n$.

Proof

We use Lemma 14. Since a decision tree of depth at most $d$ contains at most $2^{d}$ relevant variables, we can take $r=2^{d}$. We take $q=2^{4d}$. By Lemma 14, $Q(q)=O(d^{3}2^{2d}\log^{2}q)$. Then the number of membership queries is

[TABLE]

∎

A better query complexity can be obtained from the reduction in [49]. See the following Table.

The outputs of the above algorithms are the Fourier representation of the decision tree and, therefore, they are non-proper learning algorithms.

The following paper summarizes the current state of the art results in learning ${\rm DT}_{d}$

[TABLE]

7 Other Results

In this section, we give some results for learning other Boolean classes, arithmetic classes.

7.1 Other Boolean Classes

$d$-MTerm: This class is the dual of $d$-MClause. That is,

[TABLE]

Any algorithm ${\cal A}$ that learns a class $C$ can be converted to an algorithm ${\cal B}$ that learns $C^{D}$ with the same query complexity. This can be done as follows: Algorithm ${\cal B}$ runs algorithm ${\cal A}$ and for each query $a$ that ${\cal A}$ asks, algorithm ${\cal B}$ asks the query $(\overline{a_{1}},\ldots,\overline{a_{n}})$. For each answer $b$ received by the teacher, algorithm ${\cal B}$ returns the answer $\overline{b}$ to ${\cal A}$. If algorithm ${\cal A}$ outputs $h$ then algorithm ${\cal B}$ outputs

[TABLE]

$d$-Term (the dual of $d$-Clause) We first recall the definition of $(n,d)$-universal set and then show how to use it for learning $d$-Term.

A $d$-restriction problem [24, 38, 218] is a problem of the following form: Given $\Sigma=\{0,1\}$, a length $n$ and a set $B\subseteq\Sigma^{d}$ of assignments. Find a set $A\subseteq\Sigma^{n}$ of small size such that: For any $1\leq i_{1}<i_{2}<\cdots<i_{d}\leq n$ and $b\in B$ there is $a\in A$ such that $(a_{i_{1}},\ldots,a_{i_{d}})=b$.

When $B=\{0,1\}^{d}$ then $A$ is called $(n,d)$-universal set. The lower bound for the size $|A|$ of $(n,d)$-universal set is [191, 246]

[TABLE]

Using a simple probabilistic method, one can get the upper bound

[TABLE]

Also, a random uniform set of $O(2^{d}(d\log n+\log(1/\delta)))$ assignments in $\{0,1\}^{n}$ is, with probability at least $1-\delta$, $(n,d)$-universal set. The best known polynomial time (poly$(2^{d},n)$) construction for $(n,d)$-universal set is of size

[TABLE]

[218]. For $d\leq\log n/\log\log n$, a $(n,d)$-universal set of size $d^{O(1)}2^{d}\log n$ can be constructed in polynomial time [218].

Now consider the class of $d$-Term. Let ${\cal A}$ be an adaptive algorithm that learns this class. Suppose the target function is the zero term and let $S\subset\{0,1\}^{n}$ be the set of queries that the algorithm asks with this target. Then $S$ must satisfy the following property: For every $1\leq i_{1}<i_{2}<\cdots<i_{d}\leq n$ and every $b\in\{0,1\}^{d}$ there is $a\in S$ such that $(a_{i_{1}},\ldots,a_{i_{d}})=b$. Otherwise, the algorithm cannot distinguish between the zero term and the term $T=x_{i_{1}}^{b_{1}}\cdots x_{i_{d}}^{b_{d}}$ where $x^{1}=x$ and $x^{0}=\bar{x}$. This is because $T$ is also zero on all the assignments in $S$. Therefore, $S$ must be an $(n,d)$-universal set and then the query complexity of the algorithm is at least $\Omega(2^{d}\log n)$.

Now it is easy to see that any $(n,d+1)$-universal set $S$ can be used to learn non-adaptively the class $d$-Term. Just take all the positive assignments, i.e., the assignments $a\in S$ such that $f(a)=1$, and find the entries that have the same value in all of them. This uniquely determines the term. Therefore

[TABLE]

This also gives a non-adaptive learning algorithm that asks $2^{d+O(\log^{2}d)}\log n$ queries and runs in $poly(2^{d},n)$ time. Therefore the class $d$-Term is non-adaptively almost optimally learnable.

XOR: The class XOR is of size $2^{n+1}$ and therefore, by Lemma 2, any adaptive learning algorithm for XOR must ask at least $n+1$ queries. Now the trivial algorithm that asks the $n+1$ queries $\{0\}\cup\{e_{i}\}_{i=1,\ldots,n}$, where $e_{i}$ is the assignment that is $1$ in entry $i$ and zero elsewhere, learns XOR. Therefore, the class XOR is optimally learnable.

$d$-XOR: Since

[TABLE]

by Lemma 2, the lower bound for the number of queries for any randomized learning this class is $\Omega(d\log n)$. Uehara et al. gives in [266] an adaptive algorithm that learns $d$-XOR in $O(d\log n)$ queries. Therefore $d$-XOR is adaptively optimally learnable. Hofmeister gives in [159] a non-adaptive algorithm that learns $d$-XOR in $O(d\log n)$ queries. Therefore $d$-XOR is also non-adaptively optimally learnable.

$d$-Junta: The class of $d$-Juntas is studied by Damaschke in [107, 108, 109] and Bshouty and Costa in [49]. In [108], Damaschke shows that

[TABLE]

He then shows that $d$-Junta is almost optimally learnable in $d$ and efficiently learnable in $n$ [107, 109]. Using Lemma 14 with this result, we get an algorithm that asks $2^{d+o(d)}\log n$ queries and runs in time $poly(2^{d},n)$. Therefore the class $d$-Junta is almost optimally learnable. Bshouty and Costa, [49], close the above gap and showed that

[TABLE]

They also showed that randomness does not help improving the query complexity. See also other results for randomized algorithms in [49, 107], optimal algorithms for small $d$ with a constant number of rounds and bounds for the number of rounds in [49, 109].

The following is a simple adaptive learning algorithm [108]. First ask the queries of an $(n,d)$-universal set. Then take any two assignment $a$ and $b$ such that $f(a)\not=f(b)$. Then find a relevant variable by a binary search on the bits that differ between $a$ and $b$. Let $Y$ be a subset of the relevant variables that is found so far. To find another relevant variable, we search for two assignments $a$ and $b$ that give the same values for the variables in $Y$ and $f(a)\not=f(b)$. If no such assignments exist, then, $Y$ is the set of all the relevant variables and then just learn the truth table over $Y$. Otherwise, the binary search between $a$ and $b$ gives a new relevant variable. It is easy to see that the query complexity of this algorithm is $s+d\log n$ where $s$ is the size of the $(n,d)$-universal set. This shows that

[TABLE]

and, therefore, the class $d$-Junta is almost optimally adaptively learnable.

$d$-MJunta: The results in [49, 108, 109, 220] show that

[TABLE]

and

[TABLE]

Using Lemma 14 with the result of Damaschke in [109], we get a non-adaptive learning algorithm for $d$-MJunta that asks $2^{d+o(d)}\log n$ queries and runs in time $poly(2^{d},n)$. Therefore the class $d$-MJunta is almost optimally non-adaptively learnable.

The class of $n$-MJunta is studied in [154, 180] where the exact value

[TABLE]

was found. Now, by Lemma 13, $d$-MJunta is adaptively learnable in time $poly(n,$ $2^{d})$ and $O(\sqrt{d}2^{d}+d\log n)$ queries. Thus, the class $d$-MJunta is adaptively almost optimally learnable.

Decision Trees (DTd). See Section 6.

DNF: This class and its subclasses are not studied in the literature for the model of exact learning from membership queries only.

Monotone DNF: See Section 5.

CNF: The dual class of DNF.

CDNF: This class is not studied in the literature for the model of exact learning from membership queries only. Some non-optimal results can be achieved using the algorithm in [33] and the reductions in Subsection 3.4.

Monotone CDNF: The learnability of monotone CDNF is studied in [47, 110, 118]. Domingo, [110], show that the class of monotone CDNF is learnable with a polynomial number of queries in time $s^{O(\log s)}$ where $s$ is the size of the monotone CDNF. That is, the size of the MDNF and MCNF of the target. In [118] Domingo et al. study the learnability of the class Read $k$-MCDNF. This is the class of monotone CDNF functions $(f,g)$ where each variable appears at most $k$ times in its MDNF representation $f$ and any number of times in its MCNF representation $g$. See also [110] for other subclasses of monotone CDNF that are learnable from membership queries. Bshouty et al., [47], show that the class of MCDNF and $O(\log n)$-CDNF are learnable from membership queries and the NP-oracle.

Boolean Multivariate Polynomial: The efficient randomized learnability of multivariate polynomial follows from [68]. All the other algorithms in the literature require asking membership queries from an extension field. See for example [147].

XT, DFA, BMAF, ROF, BC, BF. No results are known for exact learning of those classes from membership queries only, except for the trivial result that when all the variables are relevant then ${\rm OPT}_{{\rm AD}}(C)=2^{n}$.

Boolean Halfspace (BHS): Hegedüs, [156], shows that BHS$(0,1)$ (with zero-one weights) are adaptively learnable in polynomial time with $O(n)$ queries. He also gives a lower bound $\Omega(n)$ for the number of queries. Therefore, BHS$(0,1)$ is adaptively optimally learnable. See also [266]. Hegedüs and Indyk, [163], give a non-adaptive polynomial time learning algorithm for BHS$(0,1)$ that asks $O(n^{2})$ queries.

Abboud et al., [8], show that BHS$[k]$ (Boolean Halfspaces with weights in $\{0,1,\ldots,k\}$) is constant-round learnable in $n^{O(k^{5})}$ time and queries. They also gave the lower bound $n^{\Omega(k)}$. Abasi et al. [7] give a non-adaptive algorithm for BHS$[k]$ that asks $n^{O(k^{3})}$ and a two-round algorithm that asks $n^{O(k)}$ queries and runs in time $n^{O(k)}$. Therefore, the class BHS$[k]$ is adaptively efficiently learnable.

Abboud et al. [8] give a lower bound $\Omega(2^{n}/\sqrt{n})$ for BH$(-1,0,1)$ (Boolean Halfspaces with weights $\{-1,0,+1\}$). Therefore, BH$(-1,0,1)$ is non-adaptive almost optimally learnable. Just ask all the $\{0,1\}^{n}$ queries.

Uehara et al. study some restricted classes of BHS$(0,1)$, [266].

Shevchenko and Zolotykh [261] studied halfspace function over the domain $\{0,1,\ldots,k-1\}^{n}$ when $n$ is fixed and no constraints on the coefficients. They gave the lower bound $\Omega(\log^{n-2}k)$ for learning this class from membership queries. Hegedüs [158] proves the upper bound $O(\log^{n}k/\log\log k)$. For fixed $n$, Shevchenko and Zolotykh [276] gave a polynomial time algorithm (in $\log k$) for this class. Applying Theorem 3 in [158], the upper bound $O(\log^{n-2}k)$ for the teaching dimension of a halfspace, [106], gives the upper bound $O(\log^{n-1}k/\log$ $\log k)$.

MROF. A monotone Boolean read-once formula is a monotone formula such that every input variable $x_{i}$ appears in at most one input gate. Angluin et al. gave a polynomial time algorithm that learns MROF with $O(n^{2})$ queries [20, 164]. The best lower bound for the number of queries is the information theoretic lower bound $\Omega(n\log n)$ that follows from Lemma 2.

Bshouty shows in [36] that MROF cannot be learned efficiently in parallel (poly$(\log n)$ time).

Other Classes: See classes of discrete functions and other classes in [34, 60, 146, 150, 153, 163, 261].

7.2 Classes of Arithmetic Functions

In this section, we give few results from the literature on learning arithmetic classes.

$(r,V)$-Linear Functions ($(r,V)$-LF).

The problem of learning LF is studied in [1, 77, 101, 127, 198, 199, 207, 250]. Many authors independently proved that it is optimally learnable with

[TABLE]

queries. They do not address the time complexity, although one can show that the constructions also give simple algorithms that run in polynomial time.

The class $r$-LF is studied in [37, 78, 79, 113, 145, 200, 204, 264, 266]. It is shown that

[TABLE]

Note here that in the literature they use $\log(n/r)$ to mean $\log(2n/r)$. In [37], Bshouty shows that it is optimally adaptively learnable. The problem is still open for the non-adaptive learning.

The problem of learning $(r,\Re)$-LF is studied in [54, 69, 70, 71, 83, 97, 140, 171, 179]. Bshouty and Mazzawi, [70], show that

[TABLE]

The results are derived from non-constructive probabilistic proofs. All the learning algorithms for this class are either for restricted subclasses or randomized algorithms with success probability that depend on $r$ or non-optimal.

See other subclasses in [37, 145, 225, 236]. Similar problems are studied in other areas such as coding theory [229] compressed sensing [183] Multiple Access Channels [50] (e.g., adder channels [98]) and combinatorial group testing [113, 114] (e.g., coin weighing problem [37]).

$(r,V)$-Quadratic Functions ($(r,V)$-QF).

This problem is equivalent to learning a weighted graph from additive queries, [144], where, for an additive query, one chooses a set of vertices and asks the sum of the weights of edges with both ends in the set.

The $r$-QF was studied in [1, 96, 97, 136, 144, 145, 205, 235]. The $(r,V)$-QF for different $V\subseteq\Re$ was studied in [70, 71, 72, 83, 97]. Bshouty and Mazzawi, [70], proved that

[TABLE]

The results are derived from non-constructive probabilistic proofs. For the positive real numbers $\Re^{+}$, Bshouty and Mazzawi gave in [72] a polynomial time algorithm that adaptively learns the class $(r,\Re^{+})\mbox{-}{\rm QF}$ with $O(r\log n/$ $\log r+r\log\log r)$ queries. This is the only known deterministic adaptive algorithm that runs in polynomial time. Choi, [83], gave a polynomial time randomized adaptive learning algorithm for $(r,\Re)\mbox{-}{\rm QF}$ that asks $O(r\log n/\log r)$ queries.

Bshouty and Mazzawi extended some of the above results to multilinear forms of constant degree [69].

Multivariate Polynomial: This class has been extensively studied in the literature. Ben-Or and Tiwari [75] gave the first deterministic non-adaptive polynomial time learning algorithm for sparse multivariate polynomial over a large field with an optimal number of queries. See also [141, 147, 148, 185, 193].

For identity testing and zero testing of sparse multivariate polynomials see [32, 64, 148, 155, 192, 193, 272] and references therein.

Multiplicity Automata Function: This class was first defined and studied in [44]. It is efficiently learnable from queries with a randomized MC algorithm [44].

Arithmetic Circuit and Arithmetic Formula: In [267] Valiant suggests an algebraic analog of P vs. NP, the VP vs. VNP problem. A multivariate polynomial family $\{p_{n}(x_{1},\ldots,x_{n})\}_{n\geq 1}$ is in VP if there exists a constant $c>0$ such that for all $n$, deg$(p_{n})\leq n^{c}$ and $p_{n}$ has a circuit of size bounded by $n^{c}$. Polynomial family $\{q_{n}\}_{n\geq 1}$ is in VNP if there exists a family $\{p_{n}\}\in$VP such that for every $n$

[TABLE]

Valiant shows in [267] that permanent is complete for VNP, i.e., for every polynomial family $\{q_{n}\}_{n\geq 1}$ in VNP, there is a constant $c>0$ such that for every $n\geq 1$, $q_{n}$ can be expressed as permanent of a matrix of size $n^{c}\times n^{c}$. It is believed that VP$\not=$VNP. This remains an outstanding open problem.

In [31], Agrawal and Vinay show that if there exists a deterministic polynomial time zero testing for arithmetic circuits of degree $d$ and depth $4$ then there exists a polynomial family $\{q_{n}\}_{n\geq 1}$, computable in exponential time, that is not in VP. So an efficient deterministic zero testing for such circuits leads to a proof of circuit subexponential lower bounds that may be beyond our proof techniques.

Kabanets and Impagliazzo show in [184] that even if the zero testing algorithm gets the arithmetic circuit as an input (white box) if there exists a deterministic polynomial time algorithm for zero testing for VP then either NEXP$\not\in$P/poly or VP$\not=$VNP. Therefore, any deterministic algorithm implies solving outstanding open problems in complexity. See [29, 255] for other negative results.

On the other hand, the following Schwartz-Zippel lemma, [241, 275], gives a very simple MC randomized optimal zero testing algorithm for any arithmetic circuit with a bounded degree

Lemma 23

(Schwartz-Zippel)* Let $f\in F[x_{1},\ldots,x_{n}]$ be any non-zero polynomial of degree $d$ and $S\subset F$. Then for $y_{1},\ldots,y_{n}$ selected randomly uniformly from $S$ we have*

[TABLE]

For the deterministic identity testing of arithmetic circuits of depth $3$, restricted depth $4$ circuits, circuits that compute sparse polynomials and other restricted circuits see the results in [25, 29, 30, 63, 89, 99, 188, 194, 196, 238, 239, 240, 252, 255, 256, 258, 270] and references therein. Some other results in the literature investigate the problem of minimizing the number of random bits used for identity testing. See for example [32, 64, 193].

Arithmetic Read-Once Formulas (AROF): Arithmetic Read-Once Formula is a formula where each variable appears at most once. In [59] Bshouty et al. gave an MC randomized polynomial time algorithm for AROF (with the division operation) over a large enough field $F$. In [48] Bshouty and Cleve gave a polynomial time (poly(log)) randomized parallel algorithm for this class. In [43], Bshouty and Bshouty extended the result of [59] to include the exponentiation operation. Shpilka and Volkovich in [257] gave a deterministic algorithm for learning depth $d$ AROF in time $n^{O(d)}$. In [256] Shpilka and Volkovich gave a deterministic learning algorithm for AROF that asks $n^{\log n}$ queries. They also studied the class of sum of $k$ AROFs. Recently, Volkovich gave in [214] a polynomial time algorithm for learning any AROF.

Other Classes: See other results and other classes in [1, 29, 99, 196, 232, 239, 238, 240, 256, 252, 255, 257, 258] and references therein.

8 Non-Honest Teacher

Although the aim of this survey is to summarize the results of learning from an honest teacher, we feel a need to give here some of the models of non-honest teacher and some results.

8.1 Models of Non-Honest Teacher

In this survey, the teacher model is the honest teacher model where with a query $d\in X$, the teacher answers $f(d)$.

For non-honest teacher, there are many models. One can consider a persistent teacher [27, 223] or a non-persistent teacher. For persistent teacher (or permanently faulty [223]) if the answer to the query $d$ is $y$ then no matter how many times the learner asks the same query the answer will be $y$. A non-persistent teacher is a teacher that is not persistent. In the literature the following non-honest teacher models are considered (each one can be either persistent or nonpersistent):

Incomplete Model [27]: The incomplete teacher, with a query $d$, answers $f(d)$ with probability $p$ and answers “$?$” (I DON’T KNOW) with probability $1-p$. In the persistent model, repeated queries to $d$ will give the same answer with probability $1$. In the non-adaptive model, the learner knows $p$ or some upper bound for $p$. 2. 2.

Malicious Model [186, 227, 237, 269]: (Also called random error [227] and classification noise [172]) The malicious teacher, with a query $d$, answers $f(d)$ with probability $p$ and gives an arbitrary/random wrong answer with probability $1-p$. The learner knows $p$ or some upper bound for $p$. 3. 3.

Limited Incomplete Model [22]: The limited incomplete teacher gives answers “$?$” (I DON’T KNOW) to at most $m$ queries of its choice. In the non-adaptive model, the learner knows $m$ or some upper bound for $m$. 4. 4.

Limited Malicious Model [22, 265]: (Also called the constant number of error model [16, 231]) The limited malicious teacher gives arbitrary/random wrong answers to at most $m$ queries of its choice. The learner knows $m$ or some upper bound for $m$. 5. 5.

Prefix-Bounded Error Fraction Model [222]: (Also called linearly bounded model [16]) In the adaptive model, the teacher after $t$ queries can give at most $pt$ wrong answers. In the $r$-round model, at each round with $T$ queries and for any $1\leq t\leq T$, the learner can give $pt$ wrong answers to the first $t$ queries in this round. The learner knows $p$ or some upper bound for $p$. 6. 6.

Globally Bounded Error Fraction Model [222]: In the adaptive model, if the algorithm asks $T$ queries then the teacher can give at most $pT$ wrong answers. In the $r$-round model, at each round with $T$ queries, the learner can give at most $pT$ wrong answers. The learner knows $p$ or some upper bound for $p$.

Notice that in the globally bounded error fraction model the first $pT$ queries can be all wrong while in the prefix-bounded error fraction model only $p^{2}T$ queries of the first $pT$ queries can be wrong. 7. 7.

Incomplete Prefix-Bounded Error Fraction Model: In the adaptive model, the teacher after $t$ queries can give at most $pt$ “?” answers. In the $r$-round model, at each round with $T$ queries and for any $1\leq t\leq T$, the learner can give $pt$ “?” answers to the first $t$ queries in this round. In the non-adaptive model, the learner knows $p$ or some upper bound for $p$. 8. 8.

Incomplete Globally Bounded Error Fraction Model [37]: In the adaptive model, if the algorithm asks $T$ queries then the teacher can give at most $pT$ “?” answers. In the $r$-round model, at each round with $T$ queries, the learner can give at most $pT$ “?” answers. In the non-adaptive model, the learner knows $p$ or some upper bound for $p$. 9. 9.

$E$-Sided Error Models: (Also called half-error [224], or one-sided error [231], for Boolean functions) Can be defined for any one of the above models where the wrong or“?” answers only applied when $f(d)$ is in some set $E\subset R$.

For the persistent model we define the output hypothesis to be equivalent to the target function if it agrees with the target function on all the elements of the domain except the ones for which the teacher answer “?” or gave a wrong answer.

8.2 Some Results in Learning with Non-honest Teacher

In this subsection, we give some results of learning with a non-honest teacher.

Adaptively learning Var$=\{x_{1},\ldots,x_{n}\}$ in non-honest teacher model is equivalent to the problem of “searching with lies” [265]. Ulam [265] proposed the following game. Someone thinks of a number between one and one million (which is just less than $2^{20}$). Another person is allowed to ask up to twenty questions, to each of which the first person is supposed to answer only yes or no. Obviously, the number can be guessed by asking first: is the number in the first half-million? Then again reduce the reservoir of numbers in the next question by one-half, and so on. Finally, the number is obtained in less than $\log_{2}1000000$ questions. The number $h$ corresponds to the target variable $x_{h}$ in the class Var and each question “Is $h\in H$?” corresponds to the query $(a_{1},\ldots,a_{1000000})$ where $a_{i}=1$ if and only if $i\in H$.

Ulam asked the following question: Now suppose one were allowed to lie once or twice, then how many questions would one need to get the right answer? This problem is equivalent to learning the class Var in the limited malicious model. Rényi [227] asked a similar question and therefore, the game is called Rényi-Ulam game.

This problem is completely solved with an asymptotically optimal number of queries in the limited malicious model [1, 137, 221, 245]. See also the references in [224] for results when the number of lies is small. Learning this class in two-round is studied in [102, 103, 105].

The problem is solved with an asymptotically optimal number of queries in the linearly bounded model [16, 222, 259]. It is also noted by several authors that finding a non-adaptive algorithm in this model is equivalent to constructing a $t$-error correcting code [224].

See the survey in [224] for results in other models of non-honest teacher.

For learning $d$-MClause and $s$-term $r$-MDNF with non-honest teacher see [5, 16, 80, 81, 88, 94, 95, 113, 203, 231, 271] and references therein.

9 Problems and Open Problems

In this section, we give some problems and open problems

[TABLE]

Section 1

1. 1.1.

In real life problems, the target function may change in time. Define a realistic learning model for learning functions that change in time. 2. 1.2.

In the results of this survey and almost all papers in the literature, the space complexities of learning algorithms are polynomial in ${\rm OPT}_{A}(C)$ which, for many classes $C$, is exponential in $n$ and/or other parameters that depend on the class. It is interesting to investigate learning algorithms that use small space complexity. 3. 1.3.

It is interesting to minimize the number of random bits used in randomized learning algorithms. See for example item 4.8. 4. 1.4.

It is interesting to study the exact learnability of a random function in a class $C$ from membership queries. See, for example, some models in [173, 174, 263]. 5. 1.5.

An LV randomized non-adaptive algorithm with query complexity $Q$ of complexity $T$ is an algorithm that asks at most $Q$ queries and runs in expected time $T$. So any LV randomized non-adaptive algorithm is deterministic in choosing the queries. We suggest the following definition that allows expected query complexity in non-adaptive learning algorithms: A weak LV randomized non-adaptive algorithm with complexity $T$ is a non-adaptive algorithm that (1) generates queries that are independent of the answers to the previous queries. (2) Finds the target function with probability $1$. (3) The expected number of queries is $Q$ and the expected time is $T$. 6. 1.6.

In this survey, we have shown some results for the testing problems. Some of those results are not true for LV/MC randomized algorithms. For example, in deterministic algorithms, the query complexity of non-adaptive learning is equal to the minimum size equivalent test. For randomized algorithms, one can non-adaptively equivalent test the class XOR with $O(\log 1/\delta)$ random queries whereas learning XOR by a randomized algorithm takes at least $n$ queries. It is interesting to study MC and LV randomized equivalent test and other types of tests in the adaptive and non-adaptive model. 7. 1.7.

Investigate testing in the deterministic/randomized $r$-round model. 8. 1.8.

There are very few results in the literature on parallel learning from membership queries. That is, learning in $poly(\log)$ time. Study parallel learning. 9. 1.9.

To the best of my knowledge, all the Monte Carlo learning algorithms in the literature ignore minimizing the effect of the success probability $1-\delta$ in computing the number of queries. Some of the results even ignore $\delta$ by assuming that it is constant. It is interesting to investigate the role of $\delta$ in the query complexity. 10. 1.10.

We say that a non-adaptive algorithm is strongly nonadaptive if the queries are constructed by different learners (one query for each learner) without any communication between them. It is interesting to study this model or any model with minimum communication between the learners.

Section 2

1. 2.1.

In the bound

[TABLE]

find some conditions on $C$ for which tighter bounds can be obtained. 2. 2.2.

Many lower bounds in the literature for ${\rm OPT}(C)$ are based on finding a subset of functions $C^{\prime}\subseteq C$ such that for each membership query there is an answer that eliminates at most small fraction of the functions. The best possible bound that one can get using this technique is denoted by ${\rm DEN}(C)$. In [67] Bshouty and Makhoul show that ${\rm ETD}(C)\geq{\rm DEN}(C)-1$. Find a new combinatorial measure that is a lower bound for ${\rm OPT}(C)$ and exceeds ${\rm ETD}(C)$. 3. 2.3.

The algorithm in (6) runs in time $n^{O(2^{h})}$ where $h$ is the depth of the tree. Find an algorithm with a better exponential complexity. 4. 2.4.

Find a non-adaptive learning algorithm that runs in time $poly(|X|,|C|)$ and learns $C$ using at most $(c\ln|C|){\rm OPT}_{{\rm NAD}}(C)$ queries for some $c<2$. 5. 2.5.

Study the above bounds and find approximation algorithms for randomized adaptive learning. 6. 2.6.

Study bounds and find approximation algorithms for adaptive learning of classes with small VC-dimension. 7. 2.7.

Study the above bounds and find approximation algorithms for $r$-round learning. 8. 2.8.

Is NP-oracle enough for deterministic/randomized optimal learnability? What other oracle gives learning with minimum number of membership queries? 9. 2.9.

In [53] some techniques were used in the model of exact learning from membership and equivalence queries to minimize the number of equivalence queries. Can those be used to find more query-efficient algorithms? 10. 2.10.

Study the above bounds and find approximation algorithms for classes with small extended teaching dimension. 11. 2.11.

Study bounds for LV and MC randomized algorithms.

Section 3

1. 3.1.

The reductions in subsection 3.1 are for adaptive and non-adaptive learning. It is interesting to find reduction results for $r$-round deterministic and randomized algorithms. 2. 3.2.

The reductions in subsection 3.1 are for the number of relevant variables. Find reductions for other parameters, for example, the number of terms (e.g. for MP or MDNF). 3. 3.3.

Find reductions that give algorithms that are optimally learnable or almost optimally learnable from $r$. 4. 3.4.

Lemma 15 is implicitly used for some of the results in the literature for learning some classes. For example, the Halving algorithm is an algorithm that asks equivalence query with “Majority$(C^{\prime})$” at each stage, where $C^{\prime}\subseteq C$ are the functions in $C$ that are consistent with the counterexamples seen so far. Lemma 3 is just a reduction from the Halving algorithm. It is interesting to study learnability of the classes mentioned in this survey with this technique. 5. 3.5.

There are many polynomial time exact learning algorithms from membership and equivalence queries in the literature for classes mentioned in this survey and others. See [2, 3, 17, 26, 33, 34, 35, 44, 58, 74, 182, 251, 253]. It is interesting to study the reduction of those algorithms to learning from membership queries only when some of the parameters of the class is restricted. For example, can Angluin-Frazier-Pitt learning algorithm for conjunctions of horn clauses, [17], be changed to learning from membership queries when the number of terms is bounded by $d$ or/and the size of each clause is bounded by $k$. 6. 3.6.

Let $H$ be a family of functions $h:[n]\to[q]$. For $d\leq q$ we say that $H$ is an $(n,q,d)$-perfect hash family ($(n,q,d)$-PHF) [24] if for every subset $S\subseteq[n]$ of size $|S|=d$ there is a hash function $h\in H$ such that $h|_{S}$ is injective (one-to-one) on $S$, i.e., $|h(S)|=d$. In [41] it is shown that for $q\geq 2d^{2}$. There is a $(n,q,d)$-PHF of size $O\left({d^{2}\log n}/{\log(q/d^{2})}\right)$ that can be constructed in time $O(qd^{2}n\log n/\log(q/d^{2}))$. This construction is used for many reduction in learning. It is known that there is a $(n,O(d^{2}),d)$-PHF of size $O\left({d\log n}\right)$. Finding a polynomial time construction for $(n,O(d^{2}),d)$-PHF of such size improves the query complexity of many reductions.

Section 4

1. 4.1.

Non-adaptive randomized algorithms have been proposed in [46, 66, 114, 127, 161, 165]. The following models are studied in the literature for constructing the $m\times n$ random test matrix $M$

• –

Random incidence design (RID algorithms). The entries in $M$ are chosen randomly and independently to be $1$ with probability $p$ and [math] with probability $1-p$.

• –

Random $r$-size design (RrSD algorithms). The rows in $M$ are chosen randomly and independently from the set of all vectors $\{0,1\}^{n}$ of weight $r$.

• –

Random $k$-set design (RkSD algorithms) The columns in $M$ are chosen randomly and independently from the set of all vectors $\{0,1\}^{m}$ of weight $k$.

Find lower and upper bounds for the constant in $\theta(d\log n)$ of the number of membership queries for the above non-adaptive learning algorithms. 2. 4.2.

Find a polynomial time $O(1)$-round algorithm for learning $d$-MClause that asks $O(d\log n)$ queries. 3. 4.3.

Find a deterministic non-adaptive learning algorithm for $d$-MClause that asks $O(d^{2}\log n$ $/\log d)$ queries. 4. 4.4.

A construction of a $d$-disjunct matrix is called globally explicit construction if it is deterministic polynomial time in the size of the construction. A locally explicit construction is a construction where one can find any entry in the construction in deterministic poly-log time in the size of the construction. In particular, a locally explicit construction is also globally explicit. The constructions in the literature for $d$-disjunct matrices are globally explicit constructions. Find a locally explicit construction of $d$-disjunct matrix of size $O(d^{2}\log n)$. 5. 4.5.

There are few results in the literature about learning $d$-MClause when $d$ unknown to the learner. It is interesting to study this problem. 6. 4.6.

Let ${\cal F}$ be a set of $n$ functions $f:X\to\{0,1\}$. Define $d$-MClause(${\cal F}$) the set of all functions $f_{1}\vee f_{2}\vee\cdots\vee f_{d^{\prime}}$ where $d^{\prime}\leq d$ and $f_{i}\in{\cal F}$ for all $i=1,\ldots,d^{\prime}$. Study the learnability of the class $d$-MClause(${\cal F}$). 7. 4.7.

Study the learnability of the class of monotone clauses with constant number of negated variables. 8. 4.8.

Any deterministic algorithm for non-adaptive learning $d$-MClause has query complexity $\Omega(d^{2}\log n/\log d)$ while there is a Monte Carlo non-adaptive learning algorithm that asks $O(d\log n)$ queries only and uses $O(dn\log n)$ random bits. What is the tradeoff between the number of random bits and the query complexity?

Section 5

1. 5.1.

Find strong learning algorithms for $s$-term $r$-MDNF with the parameter $r$ or/and $s$. 2. 5.2.

Many results in the literature for learning sub-classes of $s$-term $r$-MDNF are query-efficient, but are not time-efficient. It is interesting to find polynomial time learning algorithms for those classes. 3. 5.3.

Find a non-adaptive efficient learning algorithm for the class $s$-term $r$-MDNF when $r=\omega(1)$. 4. 5.4.

Angluin and Chen gave in [15] a polynomial time $5$-round Las Vegas algorithm for learning $s$-term $2$-MDNF that asks $O(s\log n+\sqrt{s}\log^{2}n)$ queries. Can this class be learned in $O(1)$-round with $O(s\log n)$ queries? 5. 5.5.

Find {\rm OPT}_{R-{\rm RAD}}($$s-term 2-MDNF) for $R=2,3,4$. 6. 5.6.

Give an optimal learning algorithm for $s$-term $r$-MDNF for constant $r>2$. 7. 5.7.

The class of Read-Once $2$-MDNF is equivalent to learning matchings [11]. Alon et al. gave bounds for deterministic, randomized and $r$-round learning this class. Extend the results to other related classes such as Read-Once $r$-MDNF, Read-Twice $2$-MDNF and Read-Once $2$-DNF.

Section 6

1. 6.1.

We show that

[TABLE]

Close the gap between the lower and upper bound. 2. 6.2.

What are the query complexities of the randomized learning algorithms for ${\rm DT}_{d}$ in [68, 251]? 3. 6.3.

The deterministic adaptive algorithm of Kushilevitz-Mansour [187] asks $O(2^{10d}$ $n\log n)$ queries. Find a more query-efficient algorithm. 4. 6.4.

Find a proper learning algorithm for ${\rm DT}_{d}$. Can ${\rm DT}_{d}$ be learned from ${\rm DT}_{poly(d)}$? 5. 6.5.

Let $X$ be a finite set and $Y$ be any set. One of the important representations of functions $f:X^{n}\to Y$ is decision tree over the alphabet $Y$ with output $X$. A decision tree over $Y$ with output $X$ is defined as follows: The constant functions $y\in Y$ are decision trees. If $f_{i}$ is a decision trees for $i=1,\ldots,t$ and $S_{1},\ldots,S_{t}$ is a partition of $X$ then, for all $j=1,\ldots,n$,

[TABLE]

is a decision tree (can also be expressed as $f^{\prime}=[x_{i}\in S_{1}]f_{1}+\cdots+[x_{i}\in S_{t}]f_{t}$. Here $[x\in S]=1$ if $x\in S$ and [math] if $x\not\in S$. Every decision tree $f^{\prime}$ can be represented as a tree $T(f^{\prime})$. If $f^{\prime}=y$ for some $y\in Y$ then $T(f^{\prime})$ is a node labeled with $y$. If $f^{\prime}$ is as in (6.5.), then $T(f^{\prime})$ has a root labeled with $x_{i}$ and has $t$ outgoing edges. The $i$th edge is labeled with $S_{i}$ and is pointing to the root of $T(f_{i})$. See for example the decision tree of tastes preference in Figure 1.

Find an efficient learning algorithm for decision trees over large alphabet. 6. 6.6.

Find an efficient deterministic non-adaptive learning algorithm for ${\rm DT}_{d}$. 7. 6.7.

Study the learnability of ${\rm DT}_{d,s}$, MDTd,s and DL.

Section 7

1. 7.1.

The randomized MC query complexity of $d$-Term is less than the deterministic query complexity. It is interesting to study $r$-round LV randomized algorithms for this class. 2. 7.2.

Close the gap between the upper bound and the lower bound of ${\rm OPT}_{{\rm NAD}}(d$-MJunta$)$. 3. 7.3.

Study the learnability of the subclasses of DNF and CDNF defined in survey. 4. 7.4.

Find ${\rm OPT}_{A}({\rm CDNF})$ and ${\rm OPT}_{A}({\rm MCDNF})$ for adaptive and non-adaptive algorithms. 5. 7.5.

Study the learnability of the classes XT, DFA and BMAF. 6. 7.6.

Find ${\rm OPT}_{{\rm NAD}}({\rm BHS}[k])$. The current upper bound is $n^{O(k^{3})}$, and the lower bound is $n^{\Omega(k)}$. 7. 7.7.

Find ${\rm OPT}_{{\rm AD}}($MROF$)$. 8. 7.8.

Study the learnability of the conjunction and disjunction of two MROF. 9. 7.9.

Find a non-adaptive algorithm for $r$-LF (resp. $r$-QF) with $O(r\log n/\omega(1))$ queries. 10. 7.10.

Find a randomized algorithm for $(r,\Re)$-LF (resp. $(r,\Re)$-QF) with an optimal number of queries with success probability $1-1/poly(n)$. 11. 7.11.

Find a deterministic efficient learning algorithm for multiplicity automata function.