Hyperedge Estimation using Polylogarithmic Subset Queries

TL;DR

This paper introduces a randomized algorithm for estimating the number of hyperedges in a hypergraph using polylogarithmic queries to a generalized independent set oracle, extending prior graph edge estimation methods.

Contribution

It generalizes existing graph edge estimation algorithms to hypergraphs with a new oracle, achieving polylogarithmic query complexity for hyperedge count approximation.

Findings

Algorithm provides $(1 \\pm \\epsilon)$-approximation of hyperedge count.

Query complexity is polylogarithmic in the number of vertices.

Applicable for constant $d$ in the generalized oracle.

Abstract

In this work, we estimate the number of hyperedges in a hypergraph ${\cal H}(U({\cal H}), {\cal F}({\cal H}))$, where $U({\cal H})$ denotes the set of vertices and ${\cal F}({\cal H}))$ denotes the set of hyperedges. We assume a query oracle access to the hypergraph ${\cal H}$. Estimating the number of edges, triangles or small subgraphs in a graph is a well studied problem. Beame \etal~and Bhattacharya \etal~gave algorithms to estimate the number of edges and triangles in a graph using queries to the {\sc Bipartite Independent Set} ({\sc BIS}) and the {\sc Tripartite Independent Set} ({\sc TIS}) oracles, respectively. We generalize the earlier works by estimating the number of hyperedges using a query oracle, known as the {\bf Generalized $d$-partite independent set oracle ({\sc GPIS})}, that takes $d$ (non-empty) pairwise disjoint subsets of vertices $A_1,\ldots,A_d \subseteq U({\cal H})$ as input, and answers whether there exists a hyperedge in ${\cal H}$ having (exactly) one vertex in each $A_i, i \in \{1,2,\ldots,d\}$. We give a randomized algorithm for the hyperedge estimation problem using the {\sc GPIS} query oracle to output $\widehat{m}$ for $m({\cal H})$ satisfying $(1-\epsilon) \cdot m({\cal H}) \leq \widehat{m} \leq (1+\epsilon) \cdot m({\cal H})$. The number of queries made by our algorithm, assuming $d$ to be a constant, is polylogarithmic in the number of vertices of the hypergraph.

Equations

$${\mathcal{R}}_d=\sum _{(c_1,\dots ,c_d)\in [k{]}^d}{m}_o\left(\chi (1,c_1),\dots ,\chi (1,c_{a_1}),\dots \dots ,\chi (s,c_{d-a_s+1}),\dots ,\chi (s,c_d)\right)\times h_d(c_1,\dots ,c_d).$$

$$\mathbb{P}\left(\left|{\mathcal{R}}_d-\frac{m_o(A_1^{[a_1]},\dots ,A_s^{[a_s]})}{k}\right|\ge 2^{2d}{\theta }^d\sqrt{d!m_o(A_1^{[a_1]},\dots ,A_s^{[a_s]}){\log }^{d}n}\right)\le p_d.$$

$$\frac{m_o(A_1,\dots ,A_d)}{8d^{d-1}{2}^d{\log }^{d-1}n}\le \hat{E}\le 20d^{d-1}{2}^d\cdot m_o(A_1,\dots ,A_d){\log }^{d-1}n$$

$$\left|\sum _{i=1}^{r^{\prime }}{w}_i^{\prime }\cdot m_o(A_{i1}^{\prime },\dots ,A_{id}^{\prime })-\sum _{i=1}^r{w}_i\cdot m_o(A_{i1},\dots ,A_{id})\right|\le \lambda S,$$

$$\mathbb{P}[h_d(\mathbf{a})=1]=\frac{1}{k}$$

$${\mathcal{R}}_d=\sum _{(c_1,\dots ,c_d)\in [k{]}^d}{m}_o\left(\chi (1,c_1),\dots ,\chi (1,c_{a_1}),\dots \dots ,\chi (s,c_{d-a_s+1}),\dots ,\chi (s,c_d)\right)\times h_d(c_1,\dots ,c_d).$$

$$\mathbb{P}\left(\left|{\mathcal{R}}_d-\frac{m_o(A_1^{[a_1]},\dots ,A_s^{[a_s]})}{k}\right|\ge 2^{2d}{\theta }^d\sqrt{d!m_o(A_1^{[a_1]},\dots ,A_s^{[a_s]}){\log }^{d}n}\right)\le p_d.$$

$$\mathbb{P}\left(\left|{\mathcal{R}}_1-\frac{m_o(A_1)}{k}\right|\ge 4\theta \cdot \sqrt{\log n\cdot m_o(A_1)}\right)\le \frac{2}{n^{16{\theta }^2}}\le p_1.$$

$${\mathbb{E}}_{{\mathcal{R}}_d}^t$$

$${\mathcal{R}}_{d-1}=\sum _{(c_1,\dots ,c_{d-1})\in [k{]}^{d-1}}{m}_o\left({\chi }^{\prime }(1,c_1),\dots ,{\chi }^{\prime }(1,c_{a_1-1}),\dots \dots ,{\chi }^{\prime }(s,c_{d-a_s})\dots {\chi }^{\prime }(s,c_{d-1})\right)\times h(c_1,\dots ,c_{d-1})$$

$$\mathbb{P}\left(\left|{\mathcal{R}}_{d-1}-\frac{m_o\left(B_1^{[a_1-1]}{B}_2^{[a_2]}\cdots B_s^{[a_s]}\right)}{k}\right|\ge 2^{2d-2}{\theta }^{d-1}\sqrt{(d-1)!m_o\left(B_1^{[a_1-1]}{B}_2^{[a_2]}\cdots B_s^{[a_s]}\right){\log }^{d-1}n}\right)\le p_{d-1}$$

$$\mathbb{P}\left(\left|{\mathcal{R}}_{d-1}-\frac{\left|{\mathcal{F}}_o^d(t,1)\right|}{k}\right|\ge 2^{2d-2}{\theta }^{d-1}\sqrt{(d-1)!{\mathcal{F}}_o^d(t,1){\log }^{d-1}n}\right)\le p_{d-1}$$

$$\mathbb{P}\left(\left|N_{\rho }^d(t,1)-N_{\nu }^d(t,1)\right|\ge 2^{2d-1}{\theta }^{d-1}\sqrt{(d-1)!{\mathcal{F}}_o(t,1){\log }^{d-1}n}\right)\le 2p_{d-1}.$$

$$\mathbb{P}\left(\left|N_{\rho }^d(t,1)-N_{\nu }^d(t,1)\right|\ge 2^{2d-1}{\theta }^{d-1}\sqrt{(d-1)!{\mathcal{F}}_o(t,1){\log }^{d-1}n}\right)$$

$${\mathbb{E}}_{{\mathcal{R}}_d}^t$$

$$\mathbb{P}\left(\left|{\mathcal{R}}_d-\mathbb{E}[{\mathcal{R}}_d]\right|>\delta +m_o(A_1^{[a_1]},\dots ,A_s^{[a_s]})\mathbb{P}(\mathcal{B})\right)\le e^{-{\delta }^2/\sum _{t=1}^n{c}_t^2}+\mathbb{P}(\mathcal{B})$$

$$\mathbb{P}\left(\left|{\mathcal{R}}_d-\frac{m_o(A_1^{[a_1]},\dots ,A_s^{[a_s]})}{k}\right|>2^{2d}{\theta }^{d-1/2}\sqrt{d!m_o(A_1^{[a_1]},\dots ,A_s^{[a_s]}){\log }^{d}n}+\frac{2d!n^d}{n^{4\theta -2d+1}}\right)$$

$${\mathbb{P}}(\mbox{{\sc Verify-Estimate} ($A_{1},\ldots,A_{d},\hat{\mathcal{R}}$) accepts})\leq\frac{1}{20\cdot 2^{d}}$$

$$\mathbb{P}\left(X_{F_o}^{\mathbf{j}}=1\right)=\prod _{i=1}^d(p(i,\mathbf{j}))\le \frac{2^{j_1}}{\hat{\mathcal{R}}}\cdot \frac{2^{j_2}}{2^{j_1}}d\log n\cdot \cdot \cdot \cdot \cdot \frac{2^{j_{d-1}}}{2^{j_{d-2}}}d\log n\cdot \frac{1}{2^{j_{d-1}}}=\frac{d^{d-2}{\log }^{d-2}n}{\hat{\mathcal{R}}}$$

$$\mathbb{P}\left(X_{\mathbf{j}}\ne 0\right)=\mathbb{P}(X_{\mathbf{j}}\ge 1)\le \mathbb{E}\left[X_{\mathbf{j}}\right]\le \frac{m_o(A_1,\dots ,A_d)}{\hat{\mathcal{R}}}{d}^{d-2}{\log }^{d-2}n$$

$$\mathbb{P}\left(X_{\mathbf{j}}\ne 0\right)\le \frac{1}{20d^{d-1}\cdot 4^d\cdot {\log }^{d-1}n}.$$

$$\displaystyle{\mathbb{P}}(\mbox{{\sc Verify-Estimate} $(A_{1},\ldots,A_{d},\hat{\mathcal{R}})$ accepts})$$

$$\left|A_1(q_1)\right|>\frac{m_o(A_1(q_1),A_2,\dots ,A_d)}{2^{q_1+1}}\ge \frac{m_o(A_1,\dots ,A_d)}{2^{q_1+1}(d\log n+1)}.$$

$$m_o(\{u_1\},\dots ,\{u_{i-1}\},A_i((Q_{i-1},U_{i-1}),q_i),\dots ,A_d)\ge \frac{m_o(\{u_1\},\dots ,\{u_{i-1}\},A_i,\dots ,A_d)}{d\log n+1}$$

$$m_o(\{u_1\},\dots ,\{u_{i-1}\},A_i((Q_{i-1},U_{i-1}),q_i),\dots ,A_d)<\left|A_i((Q_{i-1},U_i),q_i)\right|\cdot 2^{q_i+1}$$

$$\left|A_i((Q_{i-1},U_i),q_i)\right|$$

$$\mathbb{P}(\overline{{\mathcal{E}}_1})\le {\left(1-\frac{2^{q_1}}{\hat{\mathcal{R}}}\right)}^{|A_1(q_1)|}\le \exp \left(-\frac{2^{q_1}}{\hat{\mathcal{R}}}\left|A_1(q_1)\right|\right)\le \exp \left(-\frac{2^{q_1}}{\hat{\mathcal{R}}}\cdot \frac{m_o(A_1,\dots ,A_d)}{2^{q_1+1}(d\log n+1)}\right)\le \exp (-1)$$

$$\mathbb{P}\left(\overline{{\mathcal{E}}_2}|{\mathcal{E}}_1\right)\le {\left(1-\frac{2^{q_2}}{2^{q_1}}\log n\right)}^{|A_2(Q_1,U_1),q_2)|}\le \exp (-1)$$

$$\mathbb{P}\left(\overline{{\mathcal{E}}_i}|{\mathcal{E}}_1,\dots ,{\mathcal{E}}_{i-1}\right)\le {\left(1-\frac{2^{q_i}}{2^{q_{i-1}}}\log n\right)}^{|A_i(Q_{i-1},U_{i-1}),q_i)|}\le \exp (-1)$$

$$\mathbb{P}(\overline{{\mathcal{E}}_d}|{\mathcal{E}}_1,\dots ,{\mathcal{E}}_{d-1})\le {\left(1-\frac{1}{2^{q_{d-1}}}\right)}^{q_{d-1}}\le \exp (-1)$$

Full text

1 Introduction

A hypergraph $\mathcal{H}$ is a set system $(U(\mathcal{H}),\mathcal{F}(\mathcal{H}))$, where $U(\mathcal{H})$ denotes a set of $n$ vertices and $\mathcal{F}(\mathcal{H})$, a set of subsets of $U(\mathcal{H})$, denotes the set of hyperedges. A hypergraph $\mathcal{H}$ is said to be $d$-uniform if every hyperedge in $\mathcal{H}$ consists of exactly $d$ vertices. The cardinality of the hyperedge set is denoted as $m(\mathcal{H})=\left|\mathcal{F}(\mathcal{H})\right|$. We investigate the problem of estimating $m(\mathcal{H})$, where the hypergraph $\mathcal{H}$ can be accessed only by queries to an oracle that answers subset queries of a particular kind.

In the subset size estimation problem using the query model of computation, the subset query oracle is used to estimate the size of an unknown subset $S\subseteq U$, where $U$ is a known universe of elements. A subset query with a subset $T\subseteq U$ asks whether $S\cap T$ is empty or not. Viewed differently, this is about estimating an unknown set $S$ by looking at its intersection pattern with a known set $T$. At its core, a subset query essentially enquires about the existence of an intersection between two sets – a set chosen by the algorithm designer and an unknown set whose size we want to estimate. The study of subset queries was initiated in a breakthrough work by Stockmeyer [17, 18] and later formalized by Ron and Tsur [16].

Beame et al. [5] used a subset query oracle, named Bipartite Independent Set (BIS) query oracle to estimate the number of edges in a graph using polylogarithmic query complexity222query complexity means the number of queries used by the corresponding query oracle. The BIS query oracle answers a YES/NO question on the existence of an edge between two disjoint subsets of vertices of a graph $G$. Having estimated the number of edges in a graph using BIS queries, a very natural question was to estimate the number of hyperedges in a hypergraph using an appropriate query oracle. The answer to the above question is not obvious as two edges in a graph can intersect in at most one vertex but the intersection between two hyperedges in a hypergraph can be an arbitrary set. As a first step towards resolving this generalized question, Bhattacharya et al. [1, 2] considered the hyperedge estimation problem using a Tripartite Independent Set (TIS) oracle in $3$-uniform hypergraphs. A TIS query oracle takes three disjoint subsets of vertices as input and reports whether there exists a hyperedge having a vertex in each of the three sets. They showed that when the number of hyperedges having two vertices in common is bounded above (polylogarithmic in $n$), then the number of hyperedges in a $3$-uniform hypergraph can be estimated using polylogarithmic TIS queries. This leads us to ask the next set of questions given as follows.

• •

Question 1: For a $3$-uniform hypergraph, is the dependence of the TIS query complexity on the number of hyperedges with two common vertices inherent as in Bhattacharya et al. [1, 2]?

• •

Question 2: Can the subset query oracle framework of Beame et al. be extended to estimate the number of hyperedges in a $d$-uniform hypergraph using only polylogarithmic many queries?

In this paper, we give positive answers to both these questions. We show that the number of hyperedges in a $d$-uniform hypergraph can be estimated using polylogarithmic 333Here the exponent of $\log n$ is $\mathcal{O}(d)$, but the exponent of $\epsilon$ is an absolute constant. many GPIS queries. Next, we define formally our query model and state the main results.

1.1 Query model, problem description and our results

We start by defining our query oracle.

Definition 1.1**.**

Generalized $d$-partite independent set oracle (GPIS) [3]: Given $d$ pairwise disjoint subsets of vertices $A_{1},\ldots,A_{d}\subseteq U(\mathcal{H})$ of a hypergraph $\mathcal{H}$ as input, GPIS query oracle answers Yes if and only if $m(A_{1},\ldots,A_{d})\neq 0$, where $m(A_{1},\ldots,A_{d})$ denotes the number of hyperedges in $\mathcal{H}$ having exactly one vertex in each $A_{i}$, $\forall i\in\{1,2,\ldots,d\}$.

Observe that the GPIS query is a generalization of subset queries, as for $d=1$, GPIS is equivalent to asking a Yes/No question about the existence of any element of interest in the queried set. For $d=2$, GPIS is BIS. An involved use of an induction on $d$ will show how GPIS generalizes from the subset queries and this process will unravel the intricate intersection pattern of the $d$-uniform hyperedges. BIS, TIS and their generalization, the GPIS query has a transversal nature to it. A transversal [15] of a hypergraph $\mathcal{H}=(U(\mathcal{H}),\mathcal{F}(\mathcal{H}))$ is a subset $T\subseteq U(\mathcal{H})$ that intersects all sets of $\mathcal{F}(\mathcal{H})$. One can see a GPIS query as a transversal query as it answers if there exists a transversal for the disjoint subsets $A_{1},\ldots,A_{d}$ as in Definition 1.1.

We now state the precise problem that we solve in the GPIS oracle framework and present our main result in Theorem 1.2.

Hyperedge-Estimation

Input: A set of $n$ vertices $U(\mathcal{H})$ of a hypergraph $\mathcal{H}$, a GPIS oracle access to $\mathcal{H}$, and $\epsilon\in(0,1)$.

Output: An estimate $\widehat{m}$ of $m(\mathcal{H})$ such that $(1-\epsilon)\cdot m(\mathcal{H})\leq\widehat{m}\leq(1-\epsilon)\cdot m(\mathcal{H})$.

Theorem 1.2**.**

Let $\mathcal{H}$ be a hypergraph with $\left|U(\mathcal{H})\right|=n$. For any $\epsilon\in(0,1)$, Hyperedge-Estimation can be solved using ${\mathcal{O}}_{d}\left(\frac{\log^{5d+5}n}{\epsilon^{4}}\right)$ GPIS queries with high probability444high probability means a probability of at least $1-n^{-O(1)}$, where the constant in $\mathcal{O}_{d}(\cdot)$ is a function of $d$.

Recently, it came to our notice that concurrently and independently, Dell et al. [8] obtained polylogarithmic query complexity for the hyperedge estimation problem using a similar query model. We will discuss their result shortly.

1.2 Other related works

Graph parameter estimation, where one wants to estimate the number of edges, triangles or small subgraphs in a graph, is a well-studied area of research in sub-linear algorithms. Feige [12], and Goldreich and Ron [14] gave algorithms to estimate the number of edges in a graph using degree, and degree and neighbour queries, respectively. Eden et al. [10] estimated the number of triangles in a graph using degree, neighbour and edge existence queries, and gave almost matching lower bound on the query complexity. This result was generalized for estimating the number of cliques of size $k$ in [11]. Since the information revealed by degree, neighbour and edge existence queries is limited to the locality of the queried vertices, these queries are known as local queries [13]. The subset queries, used in [5, 1, 2, 8], are examples of global queries, where a global query can reveal information of the graph at a much broader level.

Goldreich and Ron [14] solved the edge estimation problem in undirected graphs using $\tilde{O}(n/\sqrt{m})$ local queries. Dell and Lapinskas [7] used the Independent set (IS) oracle to estimate the number of edges in bipartite graphs, where an IS oracle takes a subset $S$ of the vertex set as input and outputs whether $S$ is an independent set or not. Their algorithm for edge estimation in bipartite graphs makes polylogarithmic IS queries and $\mathcal{O}(n)$ edge existence queries. Beame et al. [5] extended the above result for the edge estimation problem in bipartite graphs to general graphs, and showed that the edge estimation problem in general graphs can be solved using $\tilde{\mathcal{O}}\left(\min\{\sqrt{m},n^{2}/m\}\right)$ 555$\tilde{\mathcal{O}}(\cdot)$ hides a polylogarithmic term. IS queries. Recently, Chen et al. [6] improved this result to solve the edge estimation problem using only $\tilde{\mathcal{O}}\left(\min\{\sqrt{m},n/\sqrt{m}\}\right)$ IS queries.

1.3 Setup and Notations

We denote the sets $\{1,\ldots,n\}$ and $\{0,\ldots,n\}$ by $[n]$ and $[n^{*}]$, respectively. A hypergraph $\mathcal{H}$ is a set system $(U(\mathcal{H}),\mathcal{F}(\mathcal{H}))$, where $U(\mathcal{H})$ denotes the set of vertices and $\mathcal{F}(\mathcal{H})$ denotes the set of hyperedges. The set of vertices present in a hyperedge $F\in\mathcal{F}(\mathcal{H})$ is denoted by $U(F)$ or simply $F$. A hypergraph $\mathcal{H}$ is said to be $d$-uniform if all the hyperedges in $\mathcal{H}$ consist of exactly $d$ vertices. The cardinality of the hyperedge set is $m(\mathcal{H})=\left|\mathcal{F}(\mathcal{H})\right|$. For $u\in U(\mathcal{H})$, $\mathcal{F}(u)$ denote the set of hyperedges that are incident on $u$. For $u\in U(\mathcal{H})$, the degree of $u$ in $\mathcal{H}$, denoted as $\deg_{\mathcal{H}}(u)=\left|\mathcal{F}(u)\right|$ is the number of hyperedges incident on $u$. For a set $A$ and $a\in\mathbb{N}$, $A,\ldots,A$ ($a$ times) will be denoted as $A^{[a]}$. Let $A_{1},\ldots,A_{d}\subseteq U(\mathcal{H})$ be such that for every $i,j\in[d]$ either $A_{i}=A_{j}$ or $A_{i}\cap A_{j}=\emptyset$. This has a bearing on the GPIS oracle queries we make; either the sets we query with are disjoint, or are the same. Consider the following $d$-partite sub-hypergraph of $\mathcal{H}$: $\left(U(A_{1},\ldots,A_{d}),\mathcal{F}(A_{1},\ldots,A_{d})\right)$ where the vertex set is $U(A_{1},\ldots,A_{d})=\bigcup_{i=1}^{d}A_{i}$ and the hyperedge set is $\mathcal{F}(A_{1},\ldots,A_{d})=\left\{\{i_{1},\ldots,i_{d}\}~{}|~{}i_{j}\in A_{j}\right\}$; we will denote this $d$-partite sub-hypergraph of $\mathcal{H}$ as $\mathcal{H}(A_{1},\ldots,A_{d})$. With this notation, $\mathcal{H}\left(U^{[d]}\right)$ makes sense as a $d$-partite sub-hypergraph on a vertex set $U$. The number of hyperedges in $\mathcal{H}(A_{1},\ldots,A_{d})$ is denoted by $m(A_{1},\ldots,A_{d})$.

Ordered hyperedge

We will use the subscript $o$ to denote the set of ordered hyperedges. For example, $\mathcal{F}_{o}(\mathcal{H})$ denotes the set of ordered hyperedges, $m_{o}(\mathcal{H})$ denote $\left|\mathcal{F}_{o}(\mathcal{H})\right|$, $\mathcal{F}_{o}(u)$ denote the set of ordered hyperedges incident on $u$. The ordered hyperedge set puts an order on the vertices such that $i$-th vertex of a hyperedge comes from $A_{i}$. Formally, \mathcal{F}_{o}(A_{1},\ldots,A_{d})=\{F_{o}\in\mathcal{F}_{o}(\mathcal{H}):\mbox{the i$-th vertex of$F_{o}$is in$A_{i},\forall i\in[d]}\}. The corresponding number for ordered hyperedges is $m_{o}(A_{1},\ldots,A_{d})$. We have the following relation between $m(A_{1},\ldots,A_{d})$ and $m_{o}(A_{1},\ldots,A_{d})$.

Fact 1.3**.**

For $s\in[d]$, $m_{o}(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})=m(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})~{}\times~{}\prod\limits_{i=1}^{s}a_{i}!$, where $a_{i}\in[d]$ and $\sum_{i=1}^{s}a_{i}=d$.

For a set $\cal P$, “$\cal P$ is COLORED with [n]” means that elements of $\cal P$ is assigned a color out of $[n]$ colors independently and uniformly at random. Let $\mathbb{E}[X]$ and $\mathbb{V}[X]$ denote the expectation and variance of a random variable $X$. For an event $\mathcal{E}$, $\overline{\mathcal{E}}$ denotes the complement of $\mathcal{E}$. Throughout the paper, the statement that “event $\mathcal{E}$ occurs with high probability” is equivalent to ${\mathbb{P}}(\mathcal{E})\geq 1-\frac{1}{n^{c}}$, where $c$ is an absolute constant. The statement “$a$ is an $1\pm\epsilon$ multiplicative approximation of $b$” means $\left|b-a\right|\leq\epsilon\cdot b$. For $x\in\mathbb{R}$, $\exp(x)$ denotes the standard exponential function $e^{x}$. We denote [k]\times\cdots\times[k]\mbox{~{}(p times)} using $[k]^{p}$, where $p\in\mathbb{N}$. For us, $d$ is a constant. $\mathcal{O}_{d}(\cdot)$ denotes the standard $\mathcal{O}(\cdot)$ where the constant depends on $d$. By polylogarithmic, we mean $\mathcal{O}_{d}\left(\frac{(\log n)^{\mathcal{O}_{d}(1)}}{{\epsilon}^{\mathcal{O}(1)}}\right)$. The notation $\tilde{\mathcal{O}}_{d}(\cdot)$ hides a polylogarithmic term in $\mathcal{O}_{d}(\cdot)$.

1.4 Paper organization

We define in Section 2 two other query oracles, $\mbox{{\sc GPIS}}_{1}$ and $\mbox{{\sc GPIS}}_{2}$ that can be simulated by using polylogarithmic GPIS queries. The role of these two oracles is mostly expository – they help us to describe our algorithms in a neater way. Section 3 gives a broad overview of our query algorithm that involves exact estimation, sparsification, coarse estimation and sampling. Section 3.3 contextualizes our work vis-a-vis recent works [5, 1, 2, 8]. The novel contribution of this work is sparsification which is given in detail in Section 4. Sections 5 and 6 consider the proofs for exact and coarse estimation, respectively. The algorithm and its proof of correctness are discussed in Sections 7 and 8, respectively. The equivalence proofs of the GPIS oracle and its variants, that were introduced in Section 2, are discussed in Appendix C. Some useful probability results are given in Appendix A. As this paper talks of different kinds of oracles, we have put all the definitions at one place in Appendix B for ease of reference.

2 Preliminaries: GPIS oracle and its variants

Note that the GPIS query oracle takes as input $d$ pairwise disjoint subsets of vertices. We now define two related query oracles GPIS1 and GPIS2 that remove the disjointness criteria on the input. We show that both these query oracles can be simulated by making polylogarithmic queries to the GPIS oracle with high probability. GPIS1 and GPIS2 oracles will be used in the description of the algorithm for ease of exposition.

($\mbox{{\sc GPIS}}_{1}$)

Given $s$ pairwise disjoint subsets of vertices $A_{1},\ldots,A_{s}\subseteq U(\mathcal{H})$ of a hypergraph $\mathcal{H}$ and $a_{1},\ldots,a_{s}\in[d]$ such that $\sum_{i=1}^{s}a_{i}=d$, $\mbox{{\sc GPIS}}_{1}$ query oracle on input $A_{1}^{[a_{1}]},A_{2}^{[a_{2}]},\cdots,A_{s}^{[a_{s}]}$ answers Yes if and only if $m(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})\neq 0$.

($\mbox{{\sc GPIS}}_{2}$)

Given any $d$ subsets of vertices $A_{1},\ldots,A_{d}\subseteq U(\mathcal{H})$ of a hypergraph $\mathcal{H}$, $\mbox{{\sc GPIS}}_{2}$ query oracle on input $A_{1},\ldots,A_{d}$ answers Yes if and only if $m(A_{1},\ldots,A_{d})\neq 0$.

Observe that the $\mbox{{\sc GPIS}}_{2}$ query oracle is the same as the GPIS query oracle without the requirement that the input sets are disjoint. For the $\mbox{{\sc GPIS}}_{1}$ query oracle, multiple repetitions of the same set is allowed in the input. It is obvious that a GPIS query can be simulated by a $\mbox{{\sc GPIS}}_{1}$ or $\mbox{{\sc GPIS}}_{2}$ query oracle. Using the following observations, whose proofs are given in Appendix C, we show how a $\mbox{{\sc GPIS}}_{1}$ or a $\mbox{{\sc GPIS}}_{2}$ query can be simulated by polylogarithmic GPIS queries.

Observation 2.1**.**

• (i)

A $\mbox{{\sc GPIS}}_{1}$ query can be simulated using polylogarithmic GPIS queries with high probability.

• (ii)

A $\mbox{{\sc GPIS}}_{2}$ query can be simulated using $2^{\mathcal{O}(d^{2})}$ $\mbox{{\sc GPIS}}_{1}$ queries.

• (iii)

A $\mbox{{\sc GPIS}}_{2}$ query can be simulated using polylogarithmic GPIS queries with high probability.

To prove Theorem 1.2, we first consider the following lemma. This lemma is the central result of the paper and from it, the main theorem (Theorem 1.2) follows.

Lemma 2.2**.**

Let $\mathcal{H}=(U(\mathcal{H}),\mathcal{F}(\mathcal{H}))$ be a hypergraph with $n$ vertices, i.e., $\left|U(\mathcal{H})\right|=n$. For any $\epsilon>\left({n^{-d}\log^{5d+5}n}\right)^{1/4}$, Hyperedge-Estimation can be solved with probability at least $1-\frac{1}{n^{4d}}$ and using $\mathcal{O}\left(\epsilon^{-4}\log^{5d+4}n\right)$ queries, where each query is either a $\mbox{{\sc GPIS}}_{1}$ query or a $\mbox{{\sc GPIS}}_{2}$ query.

Assuming Lemma 2.2 to be true, we now prove Theorem 1.2.

Proof of Theorem 1.2 .

If $\epsilon\leq\left({n^{-d}\log^{5d+5}n}\right)^{1/4}$, we make a GPIS query with $(\{a_{1}\},\ldots,\{a_{d}\})$ for all distinct $a_{1},\ldots,$ $a_{d}\in U(\mathcal{H})=U$ and enumerate by brute force the exact value of $m_{o}(\mathcal{H})$. So, we make at most $n^{d}=\mathcal{O}_{d}\left(\epsilon^{-4}\log^{5d+5}n\right)$ many GPIS queries as $\epsilon\leq\left({n^{-d}\log^{5d+5}n}\right)^{1/4}$. If $\epsilon>\left({n^{-d}\log^{5d+5}n}\right)^{1/4}$, we use the algorithm corresponding to Lemma 2.2, where each query is either a $\mbox{{\sc GPIS}}_{1}$ query or a $\mbox{{\sc GPIS}}_{2}$ query. However, by Observation 2.1, each $\mbox{{\sc GPIS}}_{1}$ and $\mbox{{\sc GPIS}}_{2}$ query can be simulated by $\mathcal{O}_{d}(\log n)$ many GPIS queries with high probability. So, we can replace each step of the algorithm, where we make either $\mbox{{\sc GPIS}}_{1}$ or $\mbox{{\sc GPIS}}_{2}$ query, by $\mathcal{O}_{d}(\log n)$ many GPIS queries. Hence, we are done with the proof of Theorem 1.2. ∎

In the rest of the paper, we mainly focus on proving Lemma 2.2.

3 Technical Overview

We briefly describe the overview of our work and put our work in context with recent works.

3.1 The context of our work

Beame et al. [5] developed a framework to estimate the number of edges in a graph using BIS queries. This framework involves subroutines for sparsifying a graph into a number of subgraphs each with reduced number of edges, and exactly or approximately counting the number of edges in these subgraphs. Sparsification constitutes the main building block of this framework. The sparsification routine of Beame et al. randomly colors the vertices of a graph with $2k$ colors. Let $A_{i}$ denote the set of vertices colored with color $i$. Beame et al. argued that the sum of the number of edges between $A_{i}$ and $A_{k+i}$ for $1\leq i\leq k$ approximates the total number of edges in the graph within a constant factor with high probability. Therefore, the original problem reduces to the problem of counting the number of edges in bipartite subgraphs. Here, we have $k$ bipartite subgraphs and it suffices to count the number of edges in these bipartite subgraphs. With this method, they obtained a query complexity of $O(\epsilon^{-4}\log^{14}n)$ with high probability.

Bhattacharya et al. [2] extended this framework to estimate the number of triangles in a graph using TIS queries, where given three disjoint subsets of vertices $A,\,B,\,C$, a TIS query with inputs $A,\,B,\,C$ answers whether there exists a triangle with one endpoint in each of $A,B,C$. The sparsification routine now requires to color the vertices randomly using $3k$ colors and counts the number of properly colored triangles where a properly colored triangle has one endpoint each in $A_{i}$, $A_{k+i}$, and $A_{2k+i}$ for any $1\leq i\leq k$. Unlike the scenario in Beame et al. where two edges intersect in at most one vertex, here two triangles can share an edge. Therefore, the random variables used to estimate the number of properly colored triangles are not independent. Bhattacharya et al. estimated the number of triangles assuming that the number of triangles incident on any edge is bounded by a parameter $\Delta$. In this way, they obtained a query complexity of $O(\epsilon^{-4}\Delta^{2}\log^{18}n)$.

In this paper, we fully generalize the frameworks of Stockmeyer [17, 18], Ron and Tsur [16], Beame et al. [5] and Bhattacharya et al. [2] to estimate the number of hyperedges in a $d$-uniform hypergraph using GPIS queries.

3.2 Our work in a nutshell

The algorithmic framework

In Figure 1, we give a flowchart of the algorithm. Our algorithm begins by adding the $d$-partite hypergraph $\mathcal{H}(U^{[d]})$ in a suitable data structure, let us call it $\mathcal{D}$. For each hypergraph in $\mathcal{D}$, using the exact estimation process, we exactly count the number of hyperedges in a hypergraph if the number of hyperedges is less than a threshold $\tau$ and remove the corresponding hypergraph from $\cal D$. For hypergraphs in which the number of hyperedges is more than $\tau$, we can not do exact estimation. So, we resort to sparsification if the number of hypergraphs left in $\mathcal{D}$ is not large, i.e., below a threshold $\mathcal{N}$. Hypergraph sparsification breaks a $d$-partite hypergraph into a disjoint union of $d$-partite sub-hypergraphs. As sparsification goes on in iterations, it may populate the data structure $\mathcal{D}$ with significantly many hypergraphs. In that case, we do a coarse estimation followed by sampling to have a reasonable number of sub-hypergraphs and then go back to the exact estimation step and continue in a loop. $\mbox{{\sc GPIS}}_{1}$ and $\mbox{{\sc GPIS}}_{2}$ queries will be used for exact and coarse estimations, respectively; their logarithmic equivalence with GPIS will prove the final result. For ease of analysis, we consider ordered hyperedges. Fact 1.3 relates $m(A_{1},\ldots,A_{d})$ and $m_{o}(A_{1},\ldots,A_{d})$, the number of unordered and ordered hyperedges, respectively.

Exact estimation.

In this step, we look at a $d$-partite sub-hypergraph $\mathcal{H}(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})$ and decide whether $m_{o}(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})$, the number of ordered hyperedges, is larger or smaller than a threshold $\tau$ 666Threshold $\tau$ will be fixed later in Section 5., and if $m_{o}(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})\leq\tau$, compute the exact value of $m_{o}(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})$ using Lemma 3.1, whose proof is in Section 5, and then delete it from $\mathcal{D}$. If the number of hypergraphs in $\mathcal{D}$ is below the threshold $\mathcal{N}$, with the number of ordered hyperedges in it more than $\tau$, we move to the sparsification step, else to the coarse estimation step.

Lemma 3.1** (Exact Estimation).**

There exists a deterministic algorithm $\mathcal{A}_{\text{exact}}$ that takes as input – a $d$-uniform hypergraph $\mathcal{H}$, constants $a_{1},\ldots,a_{s}\in[d]$ such that $\sum_{i=1}^{s}a_{i}=d$ where $s\in[d]$, pairwise disjoint subsets $A_{1},\ldots,A_{s}$ of $U(\mathcal{H})$, and a threshold parameter $\tau\in\mathbb{N}$ – and decides whether $m_{o}(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})\leq\tau$ using $\mathcal{O}_{d}(\tau\log n)$ $\mbox{{\sc GPIS}}_{1}$ queries. Moreover, $\mathcal{A}_{\text{exact}}$ finds the exact value of $m_{o}(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})$ when $m_{o}(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})\leq\tau$.

Sparsification.

We color $U(\mathcal{H})$, the vertices of hypergraph, with $[k]$ colors to sparsify the $d$-partite hypergraph $\mathcal{H}(U^{[d]})$ so that

• (i)

the sparsified hypergraph consists of a set of $d$-partite sub-hypergraphs and

• (ii)

a proper scaling of the sum of the number of ordered hyperedges in the sub-hypergraphs is a good estimate of $m_{o}(U^{[d]}),U=U(\mathcal{H}),$ with high probability.

The sparsification result is formally stated next. The proof uses the method of averaged bounded differences and Chernoff-Hoeffding inequality. The detailed proof is given in Section 4. The heart of our work is the general sparsification routine that we believe will find independent uses.

Lemma 3.2** (Sparsification).**

Let $\mathcal{H}$ be any $d$-uniform hypergraph and $k\geq 1$ be any positive integer. Let

• •

$h_{d}:[k]^{d}\rightarrow\{0,1\}$* be a hash function such that independently for any $d$-tuple ${\bf a}\in[k]^{d}$, ${\mathbb{P}}[h_{d}({\bf a})=1]=\frac{1}{k}$.*

• •

$A_{1},\ldots,A_{s}$* be any pairwise disjoint subsets of the vertex set $U(\mathcal{H})$, where $1\leq s\leq d$. Let us choose any $a_{i}\in[d]$ vertices from $A_{i}$ such that $\sum_{i=1}^{s}a_{i}=d$.*

• •

Vertices in $A=\bigcup_{i=1}^{s}A_{i}$ are COLORED with $[k]$. $\chi(i,j)$’s denote the color classes for each $A_{i}$, that is, \chi(i,j)=\{v\in A_{i}:~{}v~{}\mbox{is colored with color j}\}, where $i\in[s]$ and $j\in[k]$.

• •

A hyperedge $(x_{1},\ldots,x_{d})$ is said to be properly colored if $h(c_{1},\ldots,c_{d})=1$, where $c_{i}$ is the color of $x_{i}$. Let $\mathcal{R}_{d}$ denote the number of properly colored hyperedges defined as follows.

[TABLE]

Then, for a suitable constant $\theta>d$ and $p_{d}=\frac{d!}{n^{4\theta-2d}}$,

[TABLE]

Sparsification ensures that $m_{o}(U^{[d]})$, the number of ordered hyperedges between $A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]}$, is approximately preserved when $m_{o}(U^{[d]})$ is above a threshold $\tau$.

Assume that $m_{o}(U^{[d]})$ is large 777A fixed polylogarithmic quantity to be decided later in Section 7. and $\mathcal{H}(U^{[d]})$ has been sparsified. We add to the data structure $\mathcal{D}$ a set of $d$-partite sub-hypergraphs obtained from the sparsification step. Refer to the flowchart in Figure 1 again. With the new $d$-partite sub-hypergraphs, we loop back to the exact estimation step.

After the exact estimation step, we are left with some $d$-partite hypergraphs such that the number of ordered hyperedges in each hypergraph is more than the threshold $\tau$. If the number of such hypergraphs present in $\mathcal{D}$ is not large, i.e., below $\mathcal{N}$, then we sparsify each hypergraph $\mathcal{H}(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})$ using the algorithm corresponding to Lemma 3.2.

Coarse estimation and sampling.

If we have a large number of $d$-partite sub-hypergraphs of $\mathcal{H}(U^{[d]})$ and each sub-hypergraph contains a large number of ordered hyperedges, then we coarsely estimate the number of ordered hyperedges in each sub-hypergraph; see Figure 1. Our estimator is correct up to a $\mathcal{O}_{d}(\log^{d-1}n)$ factor using the algorithm corresponding to the following lemma, whose proof is given in Section 6.

Lemma 3.3** (Coarse Estimation).**

There exists an algorithm $\mathcal{A}_{\text{coarse}}$ that takes as input $d$ subsets $A_{1},\ldots,A_{d}$ of vertex set $U(\mathcal{H})$ of a $d$-uniform hypergraph $\mathcal{H}$ and returns $\hat{E}$ as an estimate for $m_{o}(A_{1},\ldots,A_{d})$ such that

[TABLE]

with probability at least $1-n^{-8d}$. Moreover, the number of $\mbox{{\sc GPIS}}_{2}$ queries made by the algorithm is $\mathcal{O}_{d}(\log^{d+1}n)$.

First, we coarsely estimate the number of ordered hyperedges in each sub-hypergraph. Then we sample a set of sub-hypergraphs such that a weighted sum of the number of hyperedges in the sample approximately preserves the sum of the number of ordered hyperedges in the sub-hypergraphs. This kind of sampling technique, also known as the importance sampling, is given for the edge estimation problem by Beame et al. [5]. The lemma corresponding to this sampling technique is formally stated as Lemma A.6 in Appendix A. The importance sampling lemma that we require is stated as follows.

Lemma 3.4** (Importance Sampling).**

Let $\{(A_{i1},\ldots,A_{id},w_{i}):i\in[r]\}$ be the set of tuples in the data structure $\mathcal{D}$ and $e_{i}$ be the coarse estimate for $m_{o}(A_{i1},\ldots,A_{id},w_{i})$ such that (i) $w_{i},e_{i}\geq 1~{}\forall i\in[r]$, (ii) $\frac{e_{i}}{\alpha}\leq m_{o}(A_{i1},\ldots,A_{id})\leq e_{i}\cdot\alpha$ for some $\alpha>0$ and $\forall~{}i\in[r]$, and (iii) $\sum_{i=1}^{r}{w_{i}\cdot m_{o}(A_{i1},\ldots,A_{id})}\leq M$. Then, there exists an algorithm that finds a set $\{(A^{\prime}_{i1},\ldots,A^{\prime}_{id},w^{\prime}_{i}):i\in[r^{\prime}]\}$ of tuples, with probability at least $1-\delta$, such that the above three conditions hold and

[TABLE]

where $S=\sum_{i=1}^{r}{w_{i}m_{o}(A_{i1},\ldots,A_{id})}$, $\lambda>0$, $\delta>0$ and $r^{\prime}=\mathcal{O}\left(\frac{\alpha^{4}\log M}{\lambda^{2}}\left(\log\log M+\log\frac{1}{\delta}\right)\right)$.

Putting things together.

The data structure $\cal D$ that our iterative algorithm uses (see the flowchart in Figure 1) has an accumulator $\Psi$ for the number of hyperedges and a set of tuples representing the $d$-partite sub-hypergraphs along with their weights – these $d$-partite sub-hypergraphs are generated from sparsification. The algorithm uses the exact estimation step (viz. Algorithm 1) to figure out if the number of hyperedges $m_{o}(A_{1},\ldots,A_{d})\leq\tau$ using Lemma 3.1. If $m_{o}(A_{1},\ldots,A_{d})\leq\tau$, we add $w\cdot m_{o}(A_{1},\ldots,A_{d})$ to $\Psi$ and remove the tuple $(A_{1},\ldots,A_{d},w)$ from $\cal D$. If the number of tuples in $\cal D$ is at most $\mathcal{N}=\kappa_{d}\cdot\frac{\log^{4d}n}{\epsilon^{2}}$, then we carry out a sparsification step, else we do an importance sampling; a coarse estimation step (viz. Algorithms 2 and 3) acts as a pre-processing step for importance sampling. We do either sparsification or coarse estimation followed by importance sampling in an iteration because their roles are complementary. While successive sparsifications ease estimating the number of hyperedges by breaking a $d$-partite hypergraph into a disjoint union of $d$-partite sub-hypergraphs, the number of such $d$-partite sub-hypergraphs may be as high as $4^{d}\cdot\mathcal{N}=4^{d}\kappa_{d}\cdot\frac{\log^{4d}n}{\epsilon^{2}}$ (see Observation 8.1). Coarse estimation followed by importance sampling (see Lemma 3.4) ensures that the effect of sparsification is neutralized so that the number of tuples in $\cal D$ is at most $\mathcal{N}$. This ensures that out of two successive iterations of the algorithm, there is at least one sparsification step. The coarse estimation applies when the number of tuples $r>\mathcal{N}=\frac{\kappa_{d}\log^{4d}n}{\epsilon^{2}}$.

For each tuple in $\cal D$, we find an estimate $\hat{E_{i}}$ using $\mathcal{O}_{d}(\log^{d+1}n)$ $\mbox{{\sc GPIS}}_{2}$ queries per tuple (see Lemma 3.3). This estimate serves as condition (ii) for the application of Lemma 3.4. Because of the importance sampling (see Lemma 3.4), one can replace the $r$ tuples in $\cal D$ with a sample of $r^{\prime}$ tuples such that the required estimate is approximately maintained. Since $r^{\prime}\leq\kappa_{d}\cdot\frac{\log^{4d}n}{\epsilon^{2}}=\mathcal{N}$, the next step will be sparsification.

Observe that the query complexity of each iteration is polylogarithmic. Note that the number of ordered hyperedges reduces by a constant factor after each sparsification step. So, the number of iterations is bounded by $\mathcal{O}_{d}(\log n)$. Hence, the query complexity of our algorithm is polylogarithmic. This completes a high level description of our algorithm.

3.3 Our work vis-a-vis some recent works

Comparison with Beame et al. [5] and Bhattacharya et al. [1, 2]:

Beame et al. [5] showed that Edge Estimation problem can be solved using $\mathcal{O}(\epsilon^{-4}\log^{14}n)$ many BIS queries. Note that a BIS query is a special case of GPIS query for $d=2$. Bhattacharya et al. [1, 2] first considered the Triangle Estimation problem in graphs using a TIS query oracle, a generalization of BIS query oracle. They showed that an $(1\pm\epsilon)$-approximation to the number of triangles can be found by using $\mathcal{O}(\Delta^{2}\epsilon^{-4}\log^{18}n)$ many TIS queries, where $\Delta$ denotes the maximum number of triangles sharing an edge. Note that the Triangle Estimation problem in graphs using TIS queries is analogous to the Hyperedge-Estimation problem in $3$-uniform hypergraphs using a special GPIS query when $d=3$. In this paper, we build on the works of Beame et al. [5] and Bhattacharya et al. [1, 2] for Hyperedge-Estimation using GPIS queries. Our main contribution in this work is twofold. First, our work can be seen as a complete generalization of the works of Beame et al.[5] and Bhattacharya et al. [1, 2] to $d$-uniform hypergraphs. Secondly, our query complexity bounds are independent of $\Delta$, this dependence on $\Delta$ was present in the work of Bhattacharya et al. [1, 2]. As mentioned earlier, the extension from the Edge Estimation problem using BIS queries to the Hyperedge Estimation problem using GPIS queries is not obvious as edges in a graph can intersect in at most one vertex, but the intersection pattern of hyperedges in hypergraphs is complicated. The sparsification algorithm in the work of Beame et al. [5] uses heavily the fact that two edges in a graph can intersect with at most one vertex. This may be the reason why Bhattacharya et al. [1, 2] required a bound on the number of triangles sharing an edge. In this paper, we show that an ingenious way of coloring the hyperedges in a $d$-uniform hypergraph and an involved use of induction on $d$ generalize the results of both Beame et al. [5] and Bhattacharya et al. [1, 2]. The main technical and non-trivial contribution in this paper is the sparsification algorithm, which enables us to generalize the works of Beame et al. [5] and Bhattacharya et al. [1, 2].

Comparison with Dell et al. [8]:

Concurrently and independently, Dell et al. [8] obtained similar results for the hyperedge estimation problem. They showed that one can find an $(1\pm\epsilon)$-approximation to the number of hyperedges in a $d$-uniform hypergraph by using $\mathcal{O}_{d}\left(\frac{\log^{4d+7}n}{\epsilon^{2}}\log\frac{1}{\delta}\right)$ colorful independent set queries with probability at least $1-\delta$. A colorful independent set query is same as that of a GPIS query oracle considered in this paper; GPIS query was introduced by Bishnu et al. [3, 4]. Note that the algorithm of Dell et al. [8] for Hyperedge-Estimation uses comparable but fewer number of GPIS queries. However, our algorithm is different and conceptually much simpler than that of Dell et al. [8]. Most importantly, the main technical tool, our sparsification step is completely different from theirs.

4 Sparsification: Proof of Lemma 3.2

We start by reproducing the statement of the Lemma 3.2 below. Sparsification ensures that the number of ordered hyperedges $m_{o}(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})$ is approximately preserved across any pairwise disjoint subsets of vertices $A_{1},\ldots,A_{s}$ and integers $a_{1},\ldots,a_{s}$ such that $\sum_{i=1}^{s}a_{i}=d$.

Lemma 4.1** (Sparsification).**

Let $\mathcal{H}$ be any $d$-uniform hypergraph and $k\geq 1$ be any positive integer. Let

• •

$h_{d}:[k]^{d}\rightarrow\{0,1\}$* be a hash function such that independently for any tuple ${\bf a}\in[k]^{d}$,*

[TABLE]

.

• •

$A_{1},\ldots,A_{s}$* be any pairwise disjoint subsets of $U(\mathcal{H})$, where $1\leq s\leq d$. Let us choose any $a_{i}\in[d]$ vertices from $A_{i}$ such that $\sum_{i=1}^{s}a_{i}=d$.*

• •

Vertices in $A=\bigcup_{i=1}^{s}A_{i}$ are COLORED with $[k]$. $\chi(i,j)$’s denote the color classes for each $A_{i}$, that is, \chi(i,j)=\{v\in A_{i}:~{}v~{}\mbox{is colored with color j}\}, where $i\in[s]$ and $j\in[k]$.

• •

A hyperedge $(x_{1},\ldots,x_{d})$ is said to be properly colored if $h(c_{1},\ldots,c_{d})=1$, where $c_{i}$ is the color of $x_{i}$. Let $\mathcal{R}_{d}$ denote the number of properly colored hyperedges defined as follows.

[TABLE]

Then, for a suitable constant $\theta>d$ and $p_{d}=\frac{d!}{n^{4\theta-2d}}$,

[TABLE]

Before getting into the formal proof of the lemma, let us explore the role of the hash function in sparsification.

4.1 The role of the hash function in sparsification

We feel our work could generalize the framework of Beame et al. [5] to hyperedge estimation mainly because of the new sparsification result aided by an involved use of induction. The sparsification result depends on the particular choice of hash function. As we move from edge to triangle and then to hyperedge in a $d$-uniform hypergraph, implicit structures blow up, and managing their interrelations also become difficult. Consider the following. Two edges can intersect in at most one vertex. two triangles can intersect in at most two vertices and two hyperedges in $d$-uniform hypergraph intersect in at most $d-1$ vertices.

To count the number of edges in $G$ using sparsification, Beame et al. [5] COLORED the vertices with $k$ colors and looked at the edges between certain pair of color classes. So, it boils down to counting the number of properly colored edges. This random coloring process can be encoded by the hash function $h:[k]\rightarrow[k]$. In our first effort in generalizing this work to count the number of triangles in a graph, we extended the above hash function to the following one – $h:[k]\times[k]\rightarrow[k]$ in  [1, 2]. Look at any edge $(a,b)$ in this graph under the above hash function. The edge is properly colored if the colors on the vertices of the edge come from specified color classes. We count all triangles on these properly colored edges, and scale it appropriately. Once an edge is not properly colored, we may miss it and hence, all triangles incident on that edge. This is why we need a bound on the number of triangles incident on an edge in [1, 2], otherwise the variance blows up and the concentration inequality gives poor result.

In this work, we consider a different hashing scheme with the hash function $h:[k]^{d}\rightarrow\{0,1\}$, in the $d$-uniform hypergraph setting, with a suitable probability of hash values becoming $1$. For triangles ($3$-uniform hypergraph), the particular hash function is thus $h:[k]^{3}\rightarrow\{0,1\}$. So, there is no concept of proper and improper edges. All edges are taken into consideration and then all triangles, and we can work with any number of triangles incident on an edge. Moreover, there is an inductive nature to this particular choice of hash function. Consider fixing the color of a vertex of a triangle. Then the hash function $h:[k]\times[k]\rightarrow\{0,1\}$ behaves as a random coloring on the other two vertices of the triangle.

4.2 Proof of the lemma

Proof.

We prove this lemma using induction on $d$.

The base case:

For $d=1$, the hash function is $h_{1}:[k]\rightarrow\{0,1\}$. The vertices in $A_{1}$ are COLORED with $[k]$. So, $\mathcal{R}_{1}=\sum\limits_{c_{1}\in[k]}m_{o}(\chi(1,c_{1}))\times h_{1}(c_{1})$. We have, $\mathbb{E}[\mathcal{R}_{1}]=\frac{m_{o}(A_{1})}{k}$ . Note that the set of $1$-uniform hyperedges $\mathcal{F}_{o}(A_{1})$ is a subset of $A_{1}$. For each hyperedge $F\in\mathcal{F}_{o}(A_{1})$, let $X_{F}$ be the indicator random variable such that $X_{F}=1$ if and only if $h_{1}(x)=1$, where the only element in $F$ gets color $x$. Therefore, ${\mathbb{P}}(X_{F}=1)=\frac{1}{k}$. Observe that the random variables $X_{F}$ are independent. Then, $\mathcal{R}_{1}=\sum\limits_{F\in\mathcal{F}_{o}(A_{1})}X_{F}$. Now, applying Hoeffding’s bound (see Lemma A.3 in Appendix A), we get

[TABLE]

The inductive case:

Let $A=\{1,\ldots,n^{\prime}\}$, where $n^{\prime}\leq n$. Let $Z_{i}\in[k]$ be the random variable that denotes the color assigned to the vertex $i\in[n^{\prime}]$. Note that $\mathcal{R}_{d}$ is a function of $Z_{1},\ldots,Z_{n^{\prime}}$, that is, $\mathcal{R}_{d}=f(Z_{1},\ldots,Z_{n^{\prime}})$. Now, consider the following definition.

Definition 4.2**.**

An ordered hyperedge $F_{o}\in\mathcal{F}_{o}\left(\chi({1,c_{1}}),\ldots,\chi({1,c_{a_{1}}}),\ldots,\chi({s,c_{d-a_{s}+1}})\ldots\chi({s,c_{d}})\right)$ is said to be properly colored if there exists $(c_{1},\ldots,c_{d})\in[k]^{d}$ such that $h_{d}(c_{1},\ldots,c_{d})=1$.

Observe that the probability that a hyperedge is properly colored is $\frac{1}{k}$, and $\mathcal{R}_{d}$, as defined in the statement of Lemma 4.1, represents the number of properly colored hyperedges. So, $\mathbb{E}[\mathcal{R}_{d}]=m_{o}(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})/k$.

Let us focus on the instance when vertices (in some arbitrary order) $1,\ldots,t-1$ have been colored and we are going to color the vertex $t$. Recall that $\mathcal{F}_{o}(t)$ denotes the set of ordered hyperedges containing $t$ as one of the vertices. Let $\mathcal{F}_{o}(t,\mu)\subseteq\mathcal{F}_{o}(t)$ be the set of ordered hyperedges containing $t$ as the $\mu$-th vertex, where $\mu\in[d]$. Note the following observation that will be used later in the proof.

Observation 4.3**.**

$\left|\mathcal{F}_{o}(t,\mu)\right|=\frac{\left|\mathcal{F}_{o}(t)\right|}{d},$ where $\mu\in[d]$.

A hyperedge $F\in\mathcal{F}_{o}(t,\mu)$ is said to be of type $\lambda$ if $F$ has exactly $\lambda$ many vertices from $[t]$, where $\lambda\in[d]$. For $\mu\in[d]$, let $\mathcal{F}_{o}^{\lambda}(t)$ and $\mathcal{F}_{o}^{\lambda}(t,\mu)$ be the set of type $\lambda$ ordered hyperedges in $\mathcal{F}_{o}(t)$ and $\mathcal{F}_{o}(t,\mu)$, respectively. Given that the vertex $t$ is colored with color $c\in[k]$, let $N_{c}^{\lambda}(t)$ and $N_{c}^{\lambda}(t,\mu)$ be the random variables that denote the number of ordered hyperedges in $\mathcal{F}_{o}^{\lambda}(t)$ and $\mathcal{F}_{o}^{\lambda}(t,\mu)$ that are properly colored, respectively.

Let $\mathbb{E}_{\mathcal{R}_{d}}^{t}$ denote the difference in the conditional expectation of the number of ordered hyperedges that are properly colored such that the $t$-th vertex is differently colored by considering the hyperedges in each $\mathcal{F}_{o}^{\lambda}(t,\mu),\lambda,\mu\in[d]$, separately.

[TABLE]

Now, consider the following claim.

Claim 4.4**.**

• (a)

${\mathbb{P}}\left(\left|N_{\rho}^{d}(t,\mu)-N_{\nu}^{d}(t,\mu)\right|\leq 2^{2d-1}\theta^{d-1}\sqrt{{(d-1)!\mathcal{F}_{o}(t,\mu)}\log^{d-1}n}\right)\geq 1-2p_{d-1}$, where $\mu\in[d]$ and $\theta>d$ is the constant mentioned in the statement of Lemma 4.1.

• (b)

$\mathbb{E}\left[N_{\rho}^{\lambda}(t)-N_{\nu}^{\lambda}(t)\right]=0,~{}\forall\lambda\in[d-1]$.

Proof of Claim 4.4.

• (a)

For simplicity, we argue for $\mu=1$. However, the argument will be similar for any $\mu\in[d]$. Consider a $(d-1)$-uniform hypergraph $\mathcal{H}^{\prime}$ such that $U(\mathcal{H}^{\prime})=[t-1]$ and $\mathcal{F}(\mathcal{H}^{\prime})=\{(x_{1},\ldots,x_{d-1}):(t,x_{1},\ldots,x_{d-1})\in\mathcal{F}_{o}^{d}(t,1)\}$. Let $h_{d-1}:[k]^{d-1}\rightarrow\{0,1\}$ be a hash function such that $h_{d-1}(x_{1},\ldots,x_{d-1})=h_{d}(t,x_{1},\ldots,x_{d})$. Observe that ${\mathbb{P}}(h_{d-1}({\bf a})=1)=\frac{1}{k}$ for each tuple ${\bf a}\in[k]^{d-1}$. Consider the $(d-1)$-partite hypergraph $\mathcal{H}^{\prime}\left(B_{1}^{[a_{1}-1]}B_{2}^{[a_{2}]}\cdots B_{s}^{[a_{s}]}\right)$, where $B_{i}=A_{i}\cap[t-1]$. Observe that $m_{o}\left(B_{1}^{[a_{1}-1]}B_{2}^{[a_{2}]}\cdots B_{s}^{[a_{s}]}\right)=\left|\mathcal{F}_{o}^{d}(t,1)\right|$. Recall that the vertices in $\cup_{i=1}^{s}A_{i}$ are COLORED with $[k]$. Let \chi^{\prime}(i,j)=\{v\in B_{i}:\mbox{v$is colored with color$j}\}. Let

[TABLE]

Observe that the random variables $N_{\rho}^{d}(t,1)$ and $N_{\nu}^{d}(t,1)$ follow the distribution of random variable $\mathcal{R}_{d-1}$. By the induction hypothesis,

[TABLE]

Using $m_{o}\left(B_{1}^{[a_{1}-1]}B_{2}^{[a_{2}]}\cdots B_{s}^{[a_{s}]}\right)=\left|\mathcal{F}_{o}^{d}(t,1)\right|$, we have

[TABLE]

Using the above equation, the claim follows using the following.

[TABLE]

Let $L=2^{2d-1}\theta^{d-1}\sqrt{{(d-1)!\mathcal{F}_{o}(t,1)}\log^{d-1}n}$. Now,

[TABLE]

• (b)

First, consider the case when $t$ is colored with color $\rho$. For $F\in\mathcal{F}_{o}^{\lambda}(t),\lambda\in[d-1]$, let $X_{F}$ be the indicator random variable such that $X_{F}=1$ if and only if $F$ is properly colored. As $F$ is of type $\lambda$, there exists at least one vertex in $F$ that is not colored yet, that is, ${\mathbb{P}}(X_{F}=1)=\frac{1}{k}$. Observe that $N^{\lambda}_{\rho}(t)=\sum\limits_{F\in\mathcal{F}_{o}^{\lambda}(t)}X_{F}$. Hence, $\mathbb{E}\left[N^{\lambda}_{\rho}(t)\right]=\left|\mathcal{F}_{o}^{\lambda}(t)\right|/k$. Similarly, one can show that $\mathbb{E}\left[N^{\lambda}_{\nu}(t)\right]=\left|\mathcal{F}_{o}^{\lambda}(t)\right|/k$. Hence, $\mathbb{E}\left[N_{\rho}^{\lambda}(t)-N_{\nu}^{\lambda}(t)\right]=0$.

∎

Now, let us come back to the proof of Lemma 4.1. By Claim 4.4 and Observation 4.3, we have the following with probability at least $1-2d\cdot p_{d-1}$.

[TABLE]

where $c_{t}=2^{2d-1}{\theta}^{d-1}\sqrt{{d!\mathcal{F}_{o}(t)}\log^{d-1}n}$.

Let $\mathcal{B}$ be the event that there exists $t\in[n]$ such that $\mathbb{E}_{\mathcal{R}^{d}}^{t}>c_{t}$. By the union bound over all $t\in[n]$, ${\mathbb{P}}(\mathcal{B})\leq 2dnp_{d-1}={2dn}\frac{{(d-1)!}}{n^{4\theta-2(d-1)}}\leq\frac{2d!}{n^{4\theta-2d+1}}$. Using the method of averaged bounded difference [9] (See Lemma A.2 in Appendix A), we have

[TABLE]

We set $\delta=2\sqrt{\theta\log n\cdot\sum\limits_{t\in[n]}c_{t}^{2}}=2^{2d}\theta^{d-1/2}\sqrt{d!m_{o}(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})\log^{d}n}$. Using $m_{o}(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})\leq n^{d}$ and ${\mathbb{P}}(\mathcal{B})\leq\frac{2d!}{n^{4\theta-4d+1}}$ , we have

[TABLE]

Assuming $n\gg d$, ${\mathbb{P}}\left(\left|\mathcal{R}_{d}-\frac{m_{o}(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})}{k}\right|\geq 2^{2d}\theta^{d}\sqrt{d!m_{o}(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})\log^{d}n}\right)\leq\frac{d!}{n^{4\theta-2d}}=p_{d}$ ∎

5 Proof of Lemma for Exact Estimation

In this Section, we prove Lemma 3.1. We start with the description of the algorithm. We restate the lemma for easy reference.

Lemma 5.1** (Exact Estimation: Lemma 3.1 restated).**

There exists a deterministic algorithm $\mathcal{A}_{\text{exact}}$ that takes as input a $d$-uniform hypergraph $\mathcal{H}$, constants $a_{1},\ldots,a_{s}\in[d]$ such that $\sum_{i=1}^{s}a_{i}=d$ where $s\in[d]$, pairwise disjoint subsets $A_{1},\ldots,A_{s}$ of $U(\mathcal{H})$, and threshold parameter $\tau\in\mathbb{N}$, and decides whether $m_{o}(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})\leq\tau$ using $\mathcal{O}_{d}(\tau\log n)$ $\mbox{{\sc GPIS}}_{1}$ queries. Moreover, the algorithm finds the exact value of $m_{o}(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})$ when $m_{o}(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})\leq\tau$.

Proof.

The algorithm first determines the exact value of $m_{o}(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})$ using $\mathcal{O}_{d}(m_{o}(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})\log n)$ $\mbox{{\sc GPIS}}_{1}$ queries. Then, we show how to modify it to obtain the desired bound. We initialize a tree ${\cal T}$ with $(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})$ as the root. The nodes of the tree are labeled with either [math] or $1$. If $m_{o}(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})=0$, we label the root with [math] and terminate. Otherwise, we label the root with $1$, and as long as there is a leaf node $(B_{1}^{[b_{1}]},\ldots,B_{t}^{[b_{t}]})$ labeled with $1$, we do the following. Here, note that $0\leq b_{i}\leq d$ and $\sum\limits_{i=1}^{t}b_{i}=d$. Also, for $i\neq j$, either $B_{i}\cap B_{j}=\emptyset$ or $B_{i}=B_{j}$.

• (i)

If $m_{o}(B_{1}^{[b_{1}]},\ldots,B_{t}^{[b_{t}]})=0$ or there exists an $i\in[t]$ such that $\left|B_{i}\right|<b_{i}$, then we label $(B_{1}^{[b_{1}]},\ldots,B_{t}^{[b_{t}]})$ with [math]. Otherwise, we label $(B_{1}^{[b_{1}]},\ldots,B_{t}^{[b_{t}]})$ with $1$.

• (ii)

We partition each $B_{i}$ into two parts $B_{i1}$ and $B_{i2}$ such that $\left|B_{i1}\right|=\lceil\frac{B_{i}}{2}\rceil$ and $\left|B_{i2}\right|=\lfloor\frac{B_{i}}{2}\rfloor$. There may exist some $B_{i}$ for which $B_{i2}=\emptyset$. We add nodes of the form $(C_{11j},\ldots,C_{1b_{1}j},\ldots,C_{t1j},\ldots,C_{tb_{t}j})$ as the children of $(B_{1}^{[b_{1}]},\ldots,B_{t}^{[b_{t}]})$, for $j\in\{1,2\}$ where $C_{ib_{i}1}=\lceil\frac{C_{ib_{i}}}{2}\rceil$ and $C_{ib_{i}2}=\lfloor\frac{C_{ib_{i}}}{2}\rfloor$. Note that for each node $(B_{1}^{[b_{1}]},\ldots,B_{t}^{[b_{t}]})$ with label $1$, we add $\prod\limits_{i=1}^{t}2^{b_{i}}=2^{d}$ nodes as children of it.

Let $\mathcal{T}^{\prime}$ be the tree after deleting all the leaf nodes in $\mathcal{T}$. Observe that $m_{o}(B_{1}^{[b_{1}]},\ldots,B_{t}^{[b_{t}]})$ is the number of leaf nodes in $\mathcal{T}^{\prime}$ and

• •

the height of ${\cal T}$ is bounded by $\max\limits_{i\in[t]}\log\left|A_{i}\right|+1\leq\log n+1$.

• •

the query complexity of the above procedure is bounded by the number of nodes in ${\cal T}$ as we make at most one query per node of $\mathcal{T}$.

The number of nodes in $\mathcal{T}^{\prime}$ which equals the number of internal nodes of $\mathcal{T}$ is bounded by $(\log n+1)m_{o}(B_{1}^{[b_{1}]},\ldots,B_{t}^{[b_{t}]})$. Hence, the number of leaf nodes in $\mathcal{T}$ is at most $2^{d}(\log n+1)m_{o}(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})$. The total number of nodes in $\mathcal{T}$ is at most $(2^{d}+1)(\log n+1)m_{o}(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})\leq 2^{d+2}m_{o}(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})\log n$. Overall, the number of $\mbox{{\sc GPIS}}_{1}$ queries made by our algorithm is at most $2^{d+2}m_{o}(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})\log n$.

The algorithm $\mathcal{A}_{\text{exact}}$ for Lemma 5.1 proceeds similar to the one presented above by initializing a tree $\mathcal{T}$ with $(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})$ as the root. The pseudo-code for $\mathcal{A}_{\text{exact}}$ is presented in Algorithm 1. If $m_{o}(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})\leq\tau$, then we can find the exact value of $m_{o}(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})$ using at most $2^{d+2}\tau\log n$ $\mbox{{\sc GPIS}}_{1}$ queries and the number of nodes in $\mathcal{T}$ is bounded by $2^{d+2}\tau\log n$. So, if the number of nodes in $\mathcal{T}$ is more than $2^{d+2}m_{o}(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})\log n$ at any instance during the execution of the algorithm, we report $m_{o}(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})>\tau$ and terminate. Hence, our algorithm makes $\mathcal{O}_{d}(\tau\log n)$ many $\mbox{{\sc GPIS}}_{1}$ queries, decides whether $m_{o}(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})\leq\tau$, and determines the exact value of $m_{o}(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})$ when $m_{o}(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})\leq\tau$. ∎

6 Proof of Lemma for Coarse Estimation

We now prove Lemma 3.3. The algorithm corresponding to Lemma 3.3 is Algorithm 3. Algorithm 2 is a subroutine in Algorithm 3. Algorithm 2 determines whether a given estimate $\hat{R}$ is correct up to $\mathcal{O}_{d}(\log^{2d-3}n)$ factor. The Lemma 6.1 and 6.2 are intermediate results needed to prove Lemma 3.3.

Lemma 6.1**.**

If $\hat{\mathcal{R}}\geq 20d^{2d-3}4^{d}m_{o}(A_{1},\ldots,A_{d})\log^{2d-3}n$, then

[TABLE]

Proof.

Consider the set of ordered hyperedges $\mathcal{F}_{o}(A_{1},\ldots,A_{d})$ in $\mathcal{H}(A_{1},\ldots,A_{d})$. Let us construct the sets $B_{i,{\bf j}}$ for $1\leq i\leq d$ as given in Algorithm 2. For an ordered hyperedge $F_{o}\in\mathcal{F}_{o}(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})$ and ${\bf j}\in\left[(d\log n)^{*}\right]^{d-1}$ 888Recall that $[n]^{*}$ denotes the set $\{0,\ldots,n\}$, let $X^{{\bf j}}_{F_{o}}$ denote the indicator random variable such that $X^{\bf j}_{F_{o}}=1$ if and only if $F_{o}\in\mathcal{F}_{o}(B_{1,{\bf j}},\ldots,B_{d,{\bf j}})$ and $X_{\bf j}=\sum\limits_{F_{o}\in\mathcal{F}_{o}(A_{1},\ldots,A_{d})}X^{\bf j}_{F_{o}}$. Note that $m_{o}(B_{1,{\bf j}},\ldots,B_{d,{\bf j}})=X_{\bf j}$. We have,

[TABLE]

Then, $\mathbb{E}\left[X_{\bf j}\right]\leq\frac{m_{o}(A_{1},\ldots,A_{d})}{\hat{\mathcal{R}}}d^{d-2}\log^{d-2}n$, and since $X_{\bf j}\geq 0$,

[TABLE]

Now, using the fact that $\hat{\mathcal{R}}\geq 20d^{2d-3}\cdot 4^{d}\cdot m_{o}(A_{1},\ldots,A_{d})\log^{2d-3}n$, we have

[TABLE]

Observe that Verify-Estimate accepts if and only if there exists ${\bf j}\in[(d\log n)^{*}]$ such that $X_{\bf j}\neq 0$. Using the union bound, we get

[TABLE]

∎

Lemma 6.2**.**

If $\hat{\mathcal{R}}\leq\frac{m_{o}(A_{1},\ldots,A_{d})}{4d\log n}$, {\mathbb{P}}(\mbox{{\sc Verify-Estimate} (A_{1},\ldots,A_{d},\hat{\mathcal{R}}) accepts})\geq\frac{1}{2^{d}}.

Proof.

First, we define some quantities and prove Claim 6.3. Then we will prove Lemma 6.2. For $q_{1}\in[(d\log n)^{*}]$, let $A_{1}(q_{1})\subseteq A_{1}$ be the set of vertices in $A_{1}$ such that for each $u_{1}\in A_{1}(q_{1})$, the number of hyperedges in $\mathcal{F}_{o}(A_{1},\ldots,A_{d})$, containing $u_{1}$ as the first vertex, lies between $2^{q_{1}}$ and $2^{q_{1}+1}-1$. For $2\leq i\leq d-1$, and $q_{j}\in[(d\log n)^{*}]~{}\forall j\in[i-1]$, consider $u_{1}\in A_{1}(q_{1}),u_{2}\in A_{2}((q_{1},u_{1}),q_{2})\ldots,$ $u_{i-1}\in A_{i-1}((q_{1},u_{1}),\ldots,(q_{i-2}u_{i-2}),q_{i-1})$. Let $A_{i}((q_{1},u_{1}),\ldots,(q_{i-1},u_{i-1}),q_{i})$ be the set of vertices in $A_{i}$ such that for each $u_{i}\in A_{i}((q_{1},u_{1}),\ldots,(q_{i-1},u_{i-1}),q_{i})$, the number of ordered hyperedges in $\mathcal{F}_{o}(A_{1},\ldots,A_{d})$, containing $u_{j}$ as the $j$-th vertex for all $j\in[i]$, lies between $2^{q_{i}}$ and $2^{q_{i}+1}-1$. We need the following result to proceed further. For ease of presentation, we use $(Q_{i},U_{i})$ to denote $(q_{1},u_{1}),\ldots,(q_{i-1},u_{i-1})$ for $2\leq i\leq d-1$. Now, we prove the following claim. It will be required to prove the lemma.

Claim 6.3**.**

• (i)

There exists $q_{1}\in[(d\log n)^{*}]$ such that $\left|A_{1}(q_{1})\right|>\frac{m_{o}(A_{1},\ldots,A_{d})}{2^{q_{1}+1}(d\log n+1)}$.

• (ii)

Let $2\leq i\leq d-1$ and $q_{j}\in[(d\log n)^{*}]~{}\forall j\in[i-1]$. Let $u_{1}\in A_{1}(q_{1})$, $u_{j}\in A_{j}((Q_{j-1},U_{j-1}),q_{j})$ $\forall j\neq 1$ and $j<i$. There exists $q_{i}\in[(d\log n)^{*}]$ such that $\left|A_{i}((Q_{i},U_{i}),q_{i})\right|>\frac{2^{q_{i-1}}}{2^{q_{i}+1}(d\log n+1)}$.

Proof.

• (i)

Observe that $m_{o}(A_{1},\ldots,A_{d})=\sum\limits_{q_{1}=0}^{d\log n}m_{o}(A_{1}(q_{1}),A_{2},\ldots,A_{d})$. So, there exists $q_{1}\in[(d\log n)^{*}]$ such that $m_{o}(A_{1}(q_{1}),A_{2},\ldots,A_{d})\geq\frac{m_{o}(A_{1},\ldots,A_{d})}{d\log n+1}$. From the definition of $A_{1}(q_{1})$, $m_{o}(A_{1}(q_{1}),A_{2},\ldots,A_{d})<\left|A_{1}(q_{1})\right|\cdot 2^{q_{1}+1}$. Hence, there exists $q_{1}\in[(d\log n)^{*}]$ such that

[TABLE]

• (ii)

Note that $m_{o}(\{u_{1}\},\ldots,\{u_{i-1}\},A_{i},\ldots,A_{d})=\sum_{q_{i}=0}^{d\log n}m_{o}(\{u_{1}\},\ldots,\{u_{i-1}\},A_{i}((Q_{i-1},U_{i-1}),q_{i}),\ldots,A_{d})$. So, there exists $q_{i}\in[(d\log n)^{*}]$ such that

[TABLE]

From the definition of $A_{i}((Q_{i-1},U_{i-1}),q_{i})$, we have

[TABLE]

Hence, there exists $q_{i}\in[(d\log n)^{*}]$ such that

[TABLE]

∎

We will be done with the proof of Lemma 6.2 by showing the following. Verify-Estimate accepts with probability at least $1/5$ when the loop variables $j_{1},\ldots,j_{d-1}$ respectively attain values $q_{1},\ldots,q_{d-1}$ such that $\left|A_{1}(q_{1})\right|>\frac{m_{o}(A_{1},\ldots,A_{d})}{2^{q_{1}+1}(d\log n+1)}$ and $\left|A_{i}((Q_{i},U_{i}),q_{i})\right|>\frac{2^{q_{i-1}}}{2^{q_{i}+1}(d\log n+1)}~{}\forall i\in[d-1]\setminus\{1\}$. The existence of such $j_{i}$’s is evident from Claim 6.3. Let ${\bf q}=(q_{1},\ldots,q_{d-1})$. Recall that $B_{i,{\bf q}}\subseteq A_{i}$ is the sample obtained when the loop variables $j_{1},\ldots,j_{d-1}$ attain values $q_{1},\ldots,q_{d-1}$, respectively. Let $\mathcal{E}_{i},i\in[d-1],$ be the events defined as follows.

• •

$\mathcal{E}_{1}~{}:~{}A_{1}(q_{1})\cap B_{1,{\bf q}}\neq\emptyset$.

• •

$\mathcal{E}_{i}~{}:~{}A_{j}((Q_{j-1},U_{j-1}),q_{j})\cap B_{j,{\bf q}}\neq\emptyset$, where $2\leq i\leq d-1$.

Observe that

[TABLE]

The last inequality uses the fact that $\hat{\mathcal{R}}\leq\frac{m_{o}(A_{1},\ldots,A_{d})}{4d\log n}$, from the condition of the lemma. Assume that $\mathcal{E}_{1}$ occurs and $u_{1}\in A_{1}(q_{1})\cap B_{1,{\bf q}}$. We will bound the probability that $A_{2}(Q_{1},U_{1}),q_{2})\cap A_{2,{\bf q}}=\emptyset$, that is $\overline{\mathcal{E}_{2}}$. Note that, by Claim 6.3 (ii), $\left|A_{2}(Q_{1},U_{1}),q_{2})\right|\geq\frac{2^{q_{1}}}{2^{q_{2}+1}(d\log n+1)}$. So,

[TABLE]

Assume that $\mathcal{E}_{1},\ldots,\mathcal{E}_{i-1}$ holds, where $3\leq i\in[d-1]$. Let $u_{1}\in A_{1}(q_{1})$ and $u_{i-1}\in A_{i-1}((Q_{i-2},U_{i-2}),q_{i-1})$. We will bound the probability that $A_{i}((Q_{i-1},U_{i-1}),q_{i})\cap B_{i,{\bf q}}=\emptyset$, that is $\overline{\mathcal{E}_{i}}$. Note that $\left|A_{i}((Q_{i-1},U_{i-1}),q_{i})\right|\geq\frac{2^{q_{i-1}}}{2^{q_{i}+1}(d\log n+1)}$. So, for $3\leq i\in[d-1]$,

[TABLE]

Assume that $\mathcal{E}_{1},\ldots,\mathcal{E}_{d-1}$ holds. Let $u_{1}\in A_{1}(q_{1})$ and $u_{i-1}\in A_{i-1}((Q_{i-2},U_{i-2}),q_{i-1})$ for all $i\in[d]\setminus\{1\}$. Let $S\subseteq A_{d}$ be the set of $d$-th vertex of the ordered hyperedges in $\mathcal{F}_{o}(A_{1},\ldots,A_{d})$ having $u_{j}$ as the $j$-th vertex for all $j\in[d-1]$. Note that $\left|S\right|\geq 2^{q_{d-1}}$. Let $\mathcal{E}_{d}$ be the event that represents the fact $S\cap B_{d,{\bf q}}\neq\emptyset$. So,

[TABLE]

Observe that Verify-Estimate accepts if $m(A_{B,{\bf q}},\ldots,B_{d,{\bf q}})\neq 0$. Also, $m(B_{1,{\bf q}},\ldots,B_{d,{\bf q}})\neq 0$ if $\bigcap\limits_{i=1}^{d}\mathcal{E}_{i}$ occurs. Hence,

[TABLE]

∎

Now, we will prove Lemma 3.3. We restate the lemma for easy reference.

Lemma 6.4** (Coarse estimation : Lemma 3.3 restated).**

There exists an algorithm $\mathcal{A}_{\text{coarse}}$ that takes as input $d$ many subsets $A_{1},\ldots,A_{d}$ of the vertex set $U(\mathcal{H})$ of a $d$-uniform hypergraph $\mathcal{H}$. The algorithm $\mathcal{A}_{\text{coarse}}$ returns $\hat{E}$ as an estimate for $m_{o}(A_{1},\ldots,A_{d})$ such that

[TABLE]

with probability $1-n^{-8d}$. Moreover, the number of $\mbox{{\sc GPIS}}_{2}$ queries made by the algorithm is $\mathcal{O}_{d}(\log^{d+1}n)$.

Proof.

Note that an execution of Coarse-Estimate for a particular $\hat{\mathcal{R}}$, repeats Verify-Estimate for $\Gamma=d\cdot 4^{d}\cdot 2000\log n$ times and gives output $\hat{\mathcal{R}}$ if more than $\frac{\Gamma}{10\cdot 2^{d}}$ many Verify-Estimate accepts. For a particular $\hat{\mathcal{R}}$, let $X_{i}$ be the indicator random variable such that $X_{i}=1$ if and only if the $i$-th execution of Verify-Estimate accepts. Also take $X=\sum_{i=1}^{\Gamma}X_{i}$. Coarse-Estimate gives output $\hat{\mathcal{R}}$ if $X>\frac{\Gamma}{10\cdot 2^{d}}$.

Consider the execution of Coarse-Estimate for a particular $\hat{\mathcal{R}}$. If $\hat{\mathcal{R}}\geq 20d^{2d-3}4^{d}\cdot m_{o}(A_{1},\ldots,A_{d})\cdot$ $\log^{2d-3}n$, we first show that Coarse-Estimate does not accept with high probability. Recall Lemma 6.1. If $\hat{\mathcal{R}}\geq 20d^{2d-3}4^{d}\cdot m_{o}(A_{1},\ldots,A_{d})\log^{2d-3}n$, ${\mathbb{P}}(X_{i}=1)\leq\frac{1}{20\cdot 2^{d}}$ and hence $\mathbb{E}[X]\leq\frac{\Gamma}{20\cdot 2^{d}}$. By using Chernoff-Hoeffding’s inequality (See Lemma A.5 (i) in Section A),

[TABLE]

Using the union bound for all $\hat{\mathcal{R}}$, the probability that Coarse-Estimate outputs some $\hat{E}=\frac{\hat{\mathcal{R}}}{d^{d-2}\cdot 2^{d}}$ such that $\hat{\mathcal{R}}\geq 20d^{2d-3}4^{d}\cdot m_{o}(A_{1},\ldots,A_{d})\log^{2d-3}n$, is at most $\frac{d\log n}{n^{10}}$. Now consider the instance when the for loop in Coarse-Estimate executes for a $\hat{\mathcal{R}}$ such that $\hat{\mathcal{R}}\leq\frac{m_{o}(A_{1},\ldots,A_{d})}{4d\log n}$. In this situation, ${\mathbb{P}}(X_{i}=1)\geq\frac{1}{2^{d}}$. So, $\mathbb{E}[X]\geq\frac{\Gamma}{2^{d}}$. By using Chernoff-Hoeffding’s inequality (See Lemma A.5 (ii) in Section A),

[TABLE]

By using the union bound for all $\hat{\mathcal{R}}$, the probability that Coarse-Estimate outputs some $\hat{E}=\frac{\hat{\mathcal{R}}}{d^{d-2}\cdot 2^{d}}$ such that $\hat{\mathcal{R}}\leq\frac{m_{o}(A_{1},\ldots,A_{d})}{4d\log n}$, is at most $\frac{d\log n}{n^{100d}}$. Observe that, the probability that Coarse-Estimate outputs some $\hat{E}=\frac{\hat{\mathcal{R}}}{d^{d-2}\cdot 2^{d}}$ such that $\hat{\mathcal{R}}\geq d^{2d-3}4^{d}m_{o}(A_{1},\ldots,A_{d})\log^{2d-3}n$ or $\hat{\mathcal{R}}\leq\frac{m_{o}(A_{1},\ldots,A_{d})}{4d\log n}$, is at most $\frac{d\log n}{n^{10d}}+\frac{d\log n}{n^{100d}}\leq\frac{1}{n^{8d}}$. Putting everything together, Coarse-Estimate gives some $\hat{E}=\frac{\hat{\mathcal{R}}}{d^{d-2}\cdot 2^{d}}$ as the output with probability at least $1-\frac{1}{n^{8d}}$ satisfying

[TABLE]

From the description of Verify-Estimate and Coarse-Estimate, the query complexity of Verify-Estimate is $\mathcal{O}(\log^{d-1}n)$ and Coarse-Estimate calls Verify-Estimate $\mathcal{O}_{d}(\log n)$ times for each choice of $\hat{R}$. Hence, Coarse-Estimate makes $\mathcal{O}_{d}(\log^{d+1}n)$ many GPIS queries. ∎

7 Algorithm

In this Section, we describe our $(1\pm\epsilon)$-approximation algorithm for the hyperedge estimation problem in hypergraph $\mathcal{H}$. When $\epsilon\leq\left({n^{-d}\log^{5d+5}n}\right)^{1/4}$, we compute the exact value of $m_{o}(\mathcal{H})$ by querying $m_{o}(\{a_{1}\},\ldots,\{a_{d}\})$ for all distinct $a_{1},\ldots,a_{d}\in U(\mathcal{H})$, and this requires only polylogarithmic number of queries. When $\epsilon>\left({n^{-d}\log^{5d+5}n}\right)^{1/4}$, we do the following. We build a data structure $\cal D$ that maintains the following two quantities.

• (i)

An accumulator $\Psi$ that builds an estimate for the number of hyperedges. We initialize $\Psi=0$.

• (ii)

A set of tuples $(A_{11},\ldots,A_{1d},w_{1}),\ldots,$ $(A_{\zeta 1},\ldots,A_{\zeta d},w_{\zeta})$ for some $\zeta>0$, where tuple $(A_{i1},\ldots,A_{id})$ corresponds to the $d$-partite sub-hypergraph $\mathcal{H}(A_{i1},\ldots,A_{id})$ and $w_{i}$ is the weight associated to $\mathcal{H}(A_{i1},\ldots,A_{id})$.

The data structure $\cal D$ is initialized with $\Psi=0$, and only one tuple $(U^{[d]},1)$. The Algorithm performs the following steps.

• (1)

Checks whether there are any tuples left in $\cal D$. If some tuples are present in $\cal D$, it goes to Step 2. Else, it outputs $\Psi$ as the estimate for the number of hyperedges.

• (2)

(Exact Counting) Fixes the threshold $\tau=\frac{k^{2}4^{2d}\theta^{2d}16d^{2}d!\log^{d+2}n}{\epsilon^{2}}$, for $k=4$999We take $k=4$ for these calculations. The argument works for any $k$.. For each tuple $(A_{1},\ldots,A_{d},w)$ in $\cal D$, it decides whether $m_{o}(A_{1},\ldots,A_{d})\leq\tau$ using Lemma 3.1. If $m_{o}(A_{1},\ldots,A_{d})\leq\tau$, it adds $wm_{o}(A_{1},\ldots,A_{d})$ to $\Psi$ and removes $(A_{1},\ldots,A_{d},w)$ from $\cal D$. If there are no tuples left in $\cal D$, it goes back to Step $1$. If the number of tuples is at most $\mathcal{N}=\kappa_{d}\cdot\frac{\log^{4d}n}{\epsilon^{2}}$, then it goes to Step $3$, else it goes to Step $4$. Note that $\kappa_{d}$ is a constant to be fixed later. By Lemma 3.1, for each tuple this step requires $\mathcal{O}_{d}(\tau\log n)=\mathcal{O}_{d}\left(\frac{\log^{d+3}n}{\epsilon^{2}}\right)$ $\mbox{{\sc GPIS}}_{1}$ queries.

• (3)

(Sparsification) For any tuple $(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]},w)$ in this step, we will have $m_{o}(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})>\tau$. Recall that $A_{i}$ and $A_{j}$ are pairwise disjoint subsets for each $1\leq i<j\leq s$. It takes a hash function $h_{d}:[k]^{d}\rightarrow\{0,1\}$ such that ${\mathbb{P}}\left(h_{d}({\bf a}=1)\right)=1/k$ independently for each tuple ${\bf a}\in[k]^{d}$. The vertices in $A=\bigcup\limits_{i=1}^{s}A_{i}$ are colored independently and uniformly at random with colors in $[k]$. It constructs sets \chi(i,j)=\{v\in A_{i}:v~{}\mbox{is colored with color j}\} for $i\in[s]$ and $j\in[k]$. It adds each tuple in $\left(\chi({1,c_{1}}),\ldots,\chi({1,c_{a_{1}}}),\ldots,\chi({s,c_{d-a_{s}+1}}),\ldots,\chi({s,c_{d}})\right)$ with weight $4w$ to $\cal D$ for which $h_{d}(c_{1},\ldots,c_{d})=1$. It removes the tuple $(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]},w)$ from $\cal D$. After processing all the tuples in this step, it goes to Step $2$. Note that no query is required in this step. The constant $4$ is obtained by putting $k=4$ in Lemma 3.2.

• (4)

(Coarse Estimation) We have $r>\mathcal{N}=\frac{\kappa_{d}\log^{4d}n}{\epsilon^{2}}$ tuples $\{(A_{i1},\ldots,A_{id},w_{i}):i\in[r]\}$ stored in the data structure $\cal D$. For each such tuple $(A_{i1},\ldots,A_{id},w_{i})$, it finds an estimate $\hat{E_{i}}$ such that $\frac{m_{o}(A_{1},\ldots,A_{d})}{82^{d}d^{d-1}\log^{d-1}n}\leq\hat{E_{i}}\leq 202^{d}d^{d-1}\log^{d-1}n~{}m_{o}(A_{1},\ldots,A_{d})$. Using Lemma 3.3, this can be done with $\mathcal{O}_{d}(\log^{d+1}n)$ $\mbox{{\sc GPIS}}_{2}$ queries per tuple. It takes a sample from the set of tuples such that the sample maintains the required estimate approximately using Lemma 3.4. It uses the algorithm for Lemma 3.4 with $\lambda=\frac{\epsilon}{4d\log n}$, $\alpha=202^{d}d^{d-1}\log^{d-1}n$ and $\delta=\frac{1}{n^{6d}}$ to find a new set $\{(A^{\prime}_{i1},\ldots,A^{\prime}_{id},w^{\prime}_{i}):i\in[r^{\prime}]\}$ of tuples satisfying the following. We will have $\left|S-\sum_{i=1}^{r^{\prime}}w^{\prime}_{i}m_{o}(A^{\prime}_{i1},\ldots,A^{\prime}_{id})\right|\leq\lambda S$ with probability $1-\frac{1}{n^{6d}}$, where $S=\sum_{i=1}^{r}w_{i}m_{o}(A_{i1},\ldots,A_{id})$. Here, $r^{\prime}\leq\kappa_{d}\cdot\frac{\log^{4d}n}{\epsilon^{2}}$, where $\kappa_{d}$ is the constant mentioned in Step $2$. It removes the set of $r$ tuples $r>\mathcal{N}$ from $\cal D$ and adds the set of $r^{\prime}$ tuples. Since the algorithm of Lemma 3.4 does not require any queries, the number of $\mbox{{\sc GPIS}}_{2}$ queries in this step in each iteration is $\mathcal{O}_{d}(\log^{d+1}n)$ per tuple.

8 Proof of Correctness

We start with the following observation for the proof of correctness.

Observation 8.1**.**

There are at most $4^{d}\cdot\mathcal{N}=4^{d}\kappa_{d}\cdot\frac{\log^{4d}n}{\epsilon^{2}}$ tuples in the data structure $\cal D$ during the execution of the algorithm.

Proof.

The number of tuples in $\cal D$ can increase by a factor of $4^{d}$ in the sparsification step. Note that we apply the sparsification step only when there are at most $\mathcal{N}=\kappa_{2}\cdot\frac{\log^{4d}n}{\epsilon^{2}}$ many tuples in $\cal D$. Hence, the number of tuples in $\cal D$ is at most $4^{d}\cdot\mathcal{N}$. ∎

Now we prove Lemma 2.2. We restate the lemma for easy reference.

Lemma 8.2** (Lemma 2.2 restated).**

If $\epsilon\geq\left({n^{-d}\log^{5d+5}n}\right)^{1/4}$, our algorithm produces $(1\pm\epsilon)$-approximation to $m_{o}(\mathcal{H})$ with probability at least $1-\frac{1}{n^{4d}}$ and makes $\mathcal{O}\left(\frac{\log^{5d+4}n}{\epsilon^{4}}\right)$ queries, where each query is either a $\mbox{{\sc GPIS}}_{1}$ query or a $\mbox{{\sc GPIS}}_{2}$ query.

To prove the above lemma, we need the following definition 8.3 along with Observations 8.4 and 8.5.

Definition 8.3**.**

Let $\mbox{{\sc Tuple}}_{i}$ be the set of tuples in the data structure $\cal D$ at the end of the $i$-th iteration. We have $\mbox{{\sc Tuple}}_{i}^{\leq\tau}=\{(A_{1},\ldots,A_{d},w):m_{o}(A_{1},\ldots,A_{d})\leq\tau\}$ and $\mbox{{\sc Tuple}}_{i}^{>\tau}=\mbox{{\sc Tuple}}_{i}\setminus\mbox{{\sc Tuple}}_{i}^{\leq\tau}$. Let $\Psi_{i}$ denote the value of $\Psi$ after the $i$-th iteration where $i\in\mathbb{N}$. The estimate for $m_{o}(\mathcal{H})=m_{o}(U^{[d]})$ after the $i$-th iteration is given as $\mbox{{\sc Est}}_{i}=\Psi_{i}+\sum\limits_{(A_{1},\ldots,A_{d},w)\in\mbox{{\sc Tuple}}_{i}}w\cdot m_{o}(A_{1},\ldots,A_{d})$. The number of active hyperedges after the $i$-th iteration is given as $\mbox{{\sc Act}}_{i}=\sum\limits_{(A_{1},\ldots,A_{d},w)\in\mbox{{\sc Tuple}}_{i}}m_{o}(A_{1},\ldots,A_{d})$.

Note that if there are some tuples left in $\cal D$ at the end of the $i$-th iteration, we do not know the value of $\mbox{{\sc Est}}_{i}$ and $\mbox{{\sc Act}}_{i}$. However, we know $\Psi_{i}$. Observe that $\Psi_{0}=0$ and $\mbox{{\sc Est}}_{0}=\mbox{{\sc Act}}_{0}=m_{o}(\mathcal{H})$.

Observation 8.4**.**

Let there be only one tuple in the data structure $\cal D$ after the $i$-th iteration for any non-negative integer $i$. Then, $\mbox{{\sc Est}}_{{i+1}}$ is a $(1+\lambda)$-approximation to $\mbox{{\sc Est}}_{i}$, where $\lambda=\frac{\epsilon}{4d\log n}$, with probability at least $1-\frac{1}{n^{5d}}$.

Proof.

From Definition 8.3,

[TABLE]

In Step $2$ of the algorithm, for each tuple $(A_{1},\ldots,A_{d},w)\in\mbox{{\sc Tuple}}_{i}^{\leq\tau}$, we determine the exact value $m_{o}(A_{1},\ldots,A_{d})$, add $w\cdot m_{o}(A_{1},\ldots,A_{d})$ to current $\Psi$ and remove the tuple from $\cal D$. Observe that

[TABLE]

If $\mbox{{\sc Tuple}}_{i}^{>\tau}$ is empty, we go to Step $1$ to report the output. Observe that in that case $\mbox{{\sc Est}}_{i+1}=\mbox{{\sc Est}}_{i}$, and we are done. If $\mbox{{\sc Tuple}}_{i}^{>\tau}$ is non-empty, then we go to either Step $3$ or Step $4$ depending on whether the number of tuples in $\cal D$ is at most $\mathcal{N}$ or more than $\mathcal{N}$, respectively, where $\mathcal{N}=\kappa_{d}\frac{\log^{4d}n}{\epsilon^{2}}$.

Case 1: (Go to Step $3$) Note that for each tuple $(A_{1},\ldots,A_{d},w)$ in $\cal D$ we have $\mbox{{\sc Tuple}}_{i}^{\geq\tau}$. We apply the sparsification step (Step $3$) for each tuple. For each tuple $(A_{1},\ldots,A_{d},w)$, we add a set of tuples $\mathcal{Z}$ by removing $(A_{1},\ldots,A_{d},w)$ from $\cal D$. By Lemma 3.2, we have the following with probability $1-\frac{1}{n^{4\theta-2d}}$.

[TABLE]

Now using $m_{o}(A_{1},\ldots,A_{d})\geq\tau=\frac{k^{2}\cdot 4^{2d}\theta^{2d}\cdot 16d^{2}\cdot d!\log^{d+2}n}{\epsilon^{2}}~{}\mbox{and}~{}k=4$ and taking $\theta=2d$, we have Equation 3 with probability $1-\frac{1}{n^{6d}}$.

[TABLE]

Since we are in Step $3$, there are at most $\mathcal{N}=\kappa_{d}\frac{\log^{4d}n}{\epsilon^{2}}$ many tuples in $\mbox{{\sc Tuple}}_{i}^{>\tau}$. As $\epsilon>\left(\frac{\log^{5d+5}}{n^{d}}\right)^{1/4}$, the probability that Equation 3 holds for each tuple in $\mbox{{\sc Tuple}}_{i}^{>\tau}$ is at least $1-\frac{1}{n^{5d}}$.

By Definition 8.3,

[TABLE]

Using Equations 2 and 3, we can show that $\mbox{{\sc Est}}_{i+1}$ is an $(1+\lambda)$-approximation to $\mbox{{\sc Est}}_{i}$, where $\lambda=\frac{\epsilon}{4d\log n}$, and the probability of success is $1-\frac{1}{n^{5d}}$.

Case $2$: (Go to Step $4$) Here, we apply coarse estimation algorithm for each tuple $(A_{1},\ldots,A_{d},w)$ present in $\cal D$ to find $\hat{E}$ such that $\frac{m_{o}(A_{1},\ldots,A_{d})}{\alpha}\leq\hat{E}\leq\alpha m_{o}(A_{1},\ldots,A_{d})$ as described in Step-4. By Lemma 3.3, the probability of success of finding the required coarse estimation for a particular tuple, is at least $1-\frac{1}{n^{8d}}.$ By Observation 8.1, we have at most $4^{d}\mathcal{N}=\frac{\kappa_{d}4^{d}\log^{4d}n}{\epsilon^{2}}$ many tuples at any instance of the algorithm. Hence, as $\epsilon>\left(\frac{\log^{5d+5}n}{n^{d}}\right)^{1/4}$, the probability that we have the desired coarse estimation for all tuples present in $\cal D$, is at least $1-\frac{1}{n^{6d}}$. We have $r>\mathcal{N}=\kappa_{d}\frac{\log^{4d}n}{\epsilon^{2}}$ many tuples in $\cal D$. Under the conditional space that we have the desired coarse estimation for all tuples present in $\cal D$, we apply the algorithm Alg corresponding to Lemma 3.4. In doing so, we get $r^{\prime}\leq\mathcal{N}$ many tuples, as described in the Step-4, with probability $1-\frac{1}{n^{6d}}$. Observe that $\mbox{{\sc Tuple}}_{i+1}$ is the set of $r^{\prime}$ tuples returned by Alg satisfying

[TABLE]

where $\lambda=\frac{\epsilon}{4d\log n}$ and $S=\sum\limits_{(A_{1},\ldots,A_{d},w)\in\mbox{{\sc Tuple}}_{i}^{>\tau}}w\cdot m_{o}(A_{1},\ldots,A_{d})$. Now, by Definition 8.3,

[TABLE]

Using Equations 2 and 4, we can show that $\mbox{{\sc Est}}_{i+1}$ is an $(1+\lambda)$-approximation to $\mbox{{\sc Est}}_{i}$ and the probability of success is $1-\left(\frac{1}{n^{6d}}+\frac{1}{n^{6d}}\right)\geq 1-\frac{1}{n^{6d-1}}$. ∎

Observation 8.5**.**

Let there be at least one tuple $(A_{1},\ldots,A_{d},w)$ in $\cal D$ after the $i$-th iteration such that $m_{o}(A_{1},\ldots,A_{d})>\tau$ for any integer $i>0$. Then, $\mbox{{\sc Act}}_{{i+2}}\leq\frac{\mbox{{\sc Act}}_{i}}{2}$ with probability at least $1-\frac{2}{n^{5d}}$.

Proof of Observation 8.5.

As there exists one tuple in $\mbox{{\sc Tuple}}_{i}^{>\tau}$, our algorithm will not terminate in Step-2. It will determine the exact values of $m_{o}(A_{1},\ldots,A_{d})$ for each $(A_{1},\ldots,A_{d},w)\in\mbox{{\sc Tuple}}_{i}^{\leq\tau}$, and then will go to either Step-3 or Step-4 depending on the cardinality of $\mbox{{\sc Tuple}}_{i}^{\leq\tau}$. By adapting the same approach as that in the proof of Observation 8.4, we can show that

• (i)

In the $(i+1)$-th iteration, if our algorithm goes to Step-3, then $\mbox{{\sc Act}}_{i+1}\leq\frac{\mbox{{\sc Act}}_{i}}{2}$ with probability $1-\frac{1}{n^{5d}}$; and

• (ii)

In the $(i+1)$-th iteration, if our algorithm goes to Step-4, then $\mbox{{\sc Act}}_{i+1}\leq{\mbox{{\sc Act}}_{i}}$ with probability $1-\frac{1}{n^{6d-1}}$.

From the description of the algorithm, it is clear that we apply sparsification either in iteration $(i+1)$ or $(i+2)$. That is, either we do sparsification in both the iterations, or we do sparsification in one iteration and coarse estimation in the other iteration, or we do sparsification in $(i+1)$-th iteration and termination of the algorithm after executing Step-2 in $(i+2)$-th iteration. Observe that in the last case, that is, if we terminate in $(i+2)$-th iteration, then $\mbox{{\sc Act}}_{i+2}=0\leq\frac{\mbox{{\sc Act}}_{i}}{2}$. In other two cases, by (i) and (ii), we have $\mbox{{\sc Act}}_{i+2}\leq\frac{\mbox{{\sc Act}}_{i}}{2}$ with probability at least $1-\frac{2}{n^{5d}}$. ∎

Now, we are ready to prove Lemma 8.2.

Proof of Lemma 8.2.

Let $i^{*}$ be the largest integer such that there exists at least one tuple $(A_{1},\ldots,A_{d},w)$ in the data structure $\cal D$ in the $i^{*}$-th iteration such that $m_{o}(A_{1},\ldots,A_{d})>\tau$. That is $\mbox{{\sc Act}}_{i^{*}}>\tau$. For ease of analysis, let us define the two following events:

$\mathcal{E}_{1}$:

$i^{*}\leq 2d\log n$.

$\mathcal{E}_{2}$:

$\mbox{{\sc Est}}_{i^{*}}$ is an $(1\pm\epsilon)$-approximation to $m_{o}(\mathcal{H})$.

Using the fact $\mbox{{\sc Act}}_{0}=m_{o}(\mathcal{H})\leq n^{d}$ along with the Observation 8.5, we have $i^{*}\leq 2d\log n$ with probability at least $1-{2d\log n}\frac{2}{n^{5d}}$. That is ${\mathbb{P}}(\mathcal{E}_{1})\geq 1-\frac{4d\log n}{n^{5d}}$.

Now let us condition on the event $\mathcal{E}_{1}$. By the definition of $i^{*}$, we do the following in the $(i^{*}+1)$-th iteration. In Step 2, for each tuple $(A_{1},\ldots,A_{d},w)$ present in $\cal D$, we determine $m_{0}(A_{1},\ldots,A_{d})$ exactly, add it to $\Psi$ and remove $(A_{1},\ldots,A_{d},w)$ from $\cal D$. Observe that $\mbox{{\sc Act}}_{i^{*}+1}=0$, that is, $\mbox{{\sc Est}}_{i^{*}+1}=\Psi_{i^{*}+1}=\mbox{{\sc Est}}_{i^{*}}$. Since there are no tuples left in $\cal D$, we go to Step $1$. At the start of the $(i^{*}+2)$-th iteration, we report $\Psi_{i^{*}+1}=\mbox{{\sc Est}}_{i^{*}}$ as the output. Using Observation 8.4, $\mbox{{\sc Est}}_{i^{*}}$ is an ${(1\pm\lambda)}^{i^{*}}$-approximation to $\mbox{{\sc Est}}_{0}$ with probability at least $1-\frac{2d\log n}{n^{5d}}$. As $\mbox{{\sc Est}}_{0}=m_{o}(\mathcal{H})$, $\lambda=\frac{\epsilon}{4d\log n}$, and $\mathcal{E}_{1}$ has occurred, we have $\mbox{{\sc Est}}_{i^{*}}$ is an $(1\pm\epsilon)$-approximation to $m_{o}(\mathcal{H})$ with probability at least $1-\frac{2d\log n}{n^{3d+1}}$. That is ${\mathbb{P}}(\mathcal{E}_{2}~{}|~{}\mathcal{E}_{1})\geq 1-\frac{2d\log n}{n^{5d}}$.

Now, we analyze the query complexity of the algorithm on the conditional space that the events $\mathcal{E}_{1}$ and $\mathcal{E}_{2}$ have occurred. By the description of the algorithm, we make $\mathcal{O}_{d}\left(\frac{\log^{d+3}n}{\epsilon^{2}}\right)$ many $\mbox{{\sc GPIS}}_{1}$ queries per tuple in Step $2$, and $\mathcal{O}_{d}(\log^{d+1}n)$ many $\mbox{{\sc GPIS}}_{2}$ queries per tuple in Step $4$. Using Observation 8.1, there can be $\mathcal{O}_{d}\left(\frac{\log^{4d}n}{\epsilon^{2}}\right)$ many tuples present in any iteration. Recall that the number of iterations is $i^{*}+2$, that is, $\mathcal{O}_{d}(\log n)$. Since $i^{*}\leq 2d\log n$, the query complexity of our algorithm is $\mathcal{O}_{d}\left(\log n\cdot\frac{\log^{4d}n}{\epsilon^{2}}\cdot\left(\frac{\log^{d+3}n}{\epsilon^{2}}+\log^{d+1}n\right)\right)=\mathcal{O}_{d}\left(\frac{\log^{5d+4}n}{\epsilon^{4}}\right)$, where each query is either a $\mbox{{\sc GPIS}}_{1}$ or a $\mbox{{\sc GPIS}}_{2}$ query.

Now we compute the probability of success of our algorithm. Observe that

${\mathbb{P}}(\mbox{{\sc Success}})\geq{\mathbb{P}}(\mathcal{E}_{1}\cap\mathcal{E}_{2})={\mathbb{P}}(\mathcal{E}_{1})\cdot{\mathbb{P}}(\mathcal{E}_{2}~{}|~{}\mathcal{E}_{1})\geq\left(1-\frac{4d\log n}{n^{5d}}\right)\cdot\left(1-\frac{2d\log n}{n^{5d}}\right)\geq 1-\frac{1}{n^{4d}}$

∎

Appendix A Some probability results

Proposition A.1**.**

Let $X$ be a random variable. Then $\mathbb{E}[X]\leq\sqrt{\mathbb{E}[X^{2}]}$.

Lemma A.2** (Theorem 7.1 in  [9]).**

Let $f$ be a function of $n$ random variables $X_{1},\ldots,X_{n}$ such that

• (i)

Each $X_{i}$ takes values from a set $A_{i}$,

• (ii)

$\mathbb{E}[f]$* is bounded, i.e., $0\leq\mathbb{E}[f]\leq M$,*

• (iii)

$\mathcal{B}$* be any event satisfying the following for each $i\in[n]$.*

[TABLE]

Then for any $\delta\geq 0$,

[TABLE]

Lemma A.3** ([9](Hoeffding’s inequality)).**

Let $X_{1},\ldots,X_{n}$ be $n$ independent random variables such that $X_{i}\in[a_{i},b_{i}]$. Then for $X=\sum\limits_{i=1}^{n}X_{i}$, the following is true for any $\delta>0$.

[TABLE]

Lemma A.4** ([9](Chernoff-Hoeffding bound)).**

Let $X_{1},\ldots,X_{n}$ be independent random variables such that $X_{i}\in[0,1]$. For $X=\sum\limits_{i=1}^{n}X_{i}$ and $\mu=\mathbb{E}[X]$, the followings hold for any $0\leq\delta\leq 1$.

[TABLE]

Lemma A.5** ([9](Chernoff-Hoeffding bound)).**

Let $X_{1},\ldots,X_{n}$ be independent random variables such that $X_{i}\in[0,1]$. For $X=\sum\limits_{i=1}^{n}X_{i}$ and $\mu_{l}\leq\mathbb{E}[X]\leq\mu_{h}$, the followings hold for any $\delta>0$.

• (i)

${\mathbb{P}}\left(X>\mu_{h}+\delta\right)\leq\exp{\left(-2\delta^{2}/n\right)}$.

• (ii)

${\mathbb{P}}\left(X<\mu_{l}-\delta\right)\leq\exp{\left(-2\delta^{2}/n\right)}$.

Lemma A.6**.**

[5*]**

Let $(D_{1},w_{1},e_{1}),\ldots,(D_{r},w_{r},e_{r})$ are the given structures and each $D_{i}$ has an associated weight ${c}(D_{i})$ satisfying*

• (i)

$w_{i},e_{i}\geq 1,\forall i\in[r]$;

• (ii)

$\frac{e_{i}}{\rho}\leq c(D_{i})\leq e_{i}\rho$* for some $\rho>0$ and all $i\in[r]$; and*

• (iii)

$\sum\limits_{i=1}^{r}{w_{i}\cdot c(D_{i})}\leq M$.

Note that the exact values $c(D_{i})$’s are not known to us. Then there exists an algorithm that finds $(D^{\prime}_{1},w^{\prime}_{1},e^{\prime}_{1}),\ldots,(D^{\prime}_{s},w^{\prime}_{s},e^{\prime}_{s})$ such that all of the above three conditions hold and $\left|\sum\limits_{i=1}^{t}{w^{\prime}_{i}\cdot c(D^{\prime}_{i})}-\sum\limits_{i=1}^{r}{w_{i}\cdot c(D_{i})}\right|\leq\lambda S$ with probability $1-\delta$; where $S=\sum\limits_{i=1}^{r}{w_{i}\cdot c(D_{i})}$ and $\lambda,\delta>0$. The time complexity of the algorithm is $\mathcal{O}(r)$ and $s=\mathcal{O}\left(\frac{\rho^{4}\log M\left(\log\log M+\log\frac{1}{\delta}\right)}{\lambda^{2}}\right).$

Appendix B Oracle definitions

Definition B.1**.**

Independent set oracle (BIS) [5]: Given a subset $X$ of the vertex set of a graph $G(V,E)$ as input, the oracle returns Yes if and only if $m(X)\neq 0$, where $m(X)$ denotes the number of edges in $G$ having both the vertices in $X$.

Definition B.2**.**

Bipartite independent set oracle (BIS) [5]: Given two disjoint subsets $A,B$ of the vertex set of a graph $G(V,E)$ as input, the oracle returns Yes if and only if $m(A,B)\neq 0$, where $m(A,B)$ denotes the number of edges having exactly one vertex in both $A$ and $B$.

Definition B.3**.**

Tripartite independent set oracle (TIS) : Given three pairwise disjoint subsets $A,B,C$ of the vertex set of a graph $G(V,E)$ as input, the oracle returns Yes if and only if $t(A,B,C)$, where $t(A,B,C)$ denotes the number of triangles in $G$ having exactly one vertex in each of $A$, $B$ and $C$.

Definition B.4**.**

Generalized $d$-partite independent set oracle (GPIS) [3]: Given $d$ pairwise disjoint subsets of vertices $A_{1},\ldots,A_{d}\subseteq U(\mathcal{H})$ of a hypergraph $\mathcal{H}$ as input, GPIS query oracle answers Yes if and only if $m(A_{1},\ldots,A_{d})\neq 0$, where $m(A_{1},\ldots,A_{d})$ denotes the number of hyperedges in $\mathcal{H}$ having exactly one vertex in each $A_{i}$, $\forall i\in\{1,2,\ldots,d\}$.

Definition B.5**.**

$\mbox{{\sc GPIS}}_{1}$ oracle: Given $s$ pairwise disjoint subsets of vertices $A_{1},\ldots,A_{s}\subseteq U(\mathcal{H})$ of a hypergraph $\mathcal{H}$ and $a_{1},\ldots,a_{s}\in[d]$ such that $\sum_{i=1}^{s}a_{i}=d$, $\mbox{{\sc GPIS}}_{1}$ query oracle on input $A_{1}^{[a_{1}]},A_{2}^{[a_{2}]},\cdots,A_{s}^{[a_{s}]}$ answers Yes if and only if $m(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})\neq 0$.

Definition B.6**.**

$\mbox{{\sc GPIS}}_{2}$ oracle: Given any $d$ subsets of vertices $A_{1},\ldots,A_{d}\subseteq U(\mathcal{H})$ of a hypergraph $\mathcal{H}$, $\mbox{{\sc GPIS}}_{2}$ query oracle on input $A_{1},\ldots,A_{d}$ answers Yes if and only if $m(A_{1},\ldots,A_{d})\neq 0$.

Appendix C GPIS query oracle and its polylogarithmic equivalents

Notice that the GPIS query oracle takes as input $d$ pairwise disjoint subsets of vertices. We now define two other query oracles GPIS1 and GPIS2 that are not as restrictive as GPIS in terms of admitting disjoint sets of vertices. We show shortly that both these oracles can be simulated by making polylogarithmic queries to GPIS with high probability. GPIS1 and GPIS2 oracles will be used for ease of exposition.

($\mbox{{\sc GPIS}}_{1}$)

Given $s$ pairwise disjoint subsets of vertices $A_{1},\ldots,A_{s}\subseteq U(\mathcal{H})$ of a hypergraph $\mathcal{H}$ and $a_{1},\ldots,a_{s}\in[d]$ such that $\sum\limits_{i=1}^{s}a_{i}=d$, $\mbox{{\sc GPIS}}_{1}$ query oracle on input $A_{1}^{[a_{1}]},A_{2}^{[a_{2}]},\cdots,A_{s}^{[a_{s}]}$ answers Yes if and only if $m(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})\neq 0$.

($\mbox{{\sc GPIS}}_{2}$)

Given any $d$ subsets of vertices $A_{1},\ldots,A_{d}\subseteq U(\mathcal{H})$ of a hypergraph $\mathcal{H}$, $\mbox{{\sc GPIS}}_{2}$ query oracle on input $A_{1},\ldots,A_{d}$ answers Yes if and only if $m(A_{1},\ldots,A_{d})\neq 0$.

If from GPIS, we rule out the pairwise disjoint subset condition on $A_{1},\ldots,A_{d}$, we get $\mbox{{\sc GPIS}}_{2}$ . In $\mbox{{\sc GPIS}}_{1}$ , we allow multiple repetitions of the same set.

From the above definitions, it is clear that a GPIS query can be simulated by a $\mbox{{\sc GPIS}}_{1}$ or $\mbox{{\sc GPIS}}_{2}$ query. Through the following observations, we show how a $\mbox{{\sc GPIS}}_{1}$ or a $\mbox{{\sc GPIS}}_{2}$ query can be simulated by polylogarithmic many GPIS queries.

Observation C.1**.**

• (i)

A $\mbox{{\sc GPIS}}_{1}$ query can be simulated by using polylogarithmic GPIS queries with high probability.

• (ii)

A $\mbox{{\sc GPIS}}_{2}$ query can be simulated using $2^{\mathcal{O}(d^{2})}$ $\mbox{{\sc GPIS}}_{1}$ queries.

• (iii)

A $\mbox{{\sc GPIS}}_{2}$ query can be simulated using polylogarithmic GPIS queries with high probability.

Proof.

• (i)

Let the input of $\mbox{{\sc GPIS}}_{1}$ query oracle be $A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]}$ such that $a_{i}\in[d]~{}\forall i\in[s]$ and $\sum\limits_{i=1}^{s}a_{i}=d$. For each $i\in[s]$, we partition $A_{i}$ (only one copy of $A_{i}$, and not $a_{i}$ many copies of $A_{i}$) randomly into $a_{i}$ parts, let $\{B_{i}^{j}:j\in[a_{i}]\}$ be the resulting partition of $A_{i}$. Then we make a GPIS query with input $B_{1}^{1},\ldots,B_{1}^{a_{1}},\ldots,B_{s}^{1},\ldots,B_{s}^{a_{s}}$. Note that

[TABLE]

So, if $\mbox{{\sc GPIS}}_{1}$ outputs ‘No’ to query $A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]}$, then the above GPIS query will also report ‘No’ as its answer. If $\mbox{{\sc GPIS}}_{1}$ answers ‘Yes’, then consider a particular hyperedge $F\in\mathcal{F}(A_{1}^{[a_{1}]},\ldots,A_{s}^{[a_{s}]})$. Observe that

[TABLE]

We can boost up the success probability arbitrarily by repeating the above procedure polylogarithmic many times.

• (ii)

Let the input to $\mbox{{\sc GPIS}}_{2}$ query oracle be $A_{1},\ldots,A_{d}$. Let us partition each set $A_{i}$ into at most $2^{d-1}-1$ subsets depending on $A_{i}$’s intersection with $A_{j}$’s for $j\neq i$. Let $\mathcal{P}_{i}$ denote the corresponding partition of $A_{i}$, $i\in[d]$. Observe that for any $i\neq j$, if we take any $B_{i}\in\mathcal{P}_{i}$ and $B_{j}\in\mathcal{P}_{j}$, then either $B_{i}=B_{j}$ or $B_{i}\cap B_{j}=\emptyset$.

For each $(B_{1},\ldots,B_{d})\in\mathcal{P}_{1}\times\ldots\times\mathcal{P}_{d}$, we make a $\mbox{{\sc GPIS}}_{1}$ query with input $(B_{1},\ldots,B_{d})$. Total number of such $\mbox{{\sc GPIS}}_{1}$ queries is at most $2^{\mathcal{O}(d^{2})}$, and we report ‘Yes’ to the $\mbox{{\sc GPIS}}_{2}$ query if and only if at least one $\mbox{{\sc GPIS}}_{1}$ query, out of the $2^{\mathcal{O}(d^{2})}$ queries, reports ‘Yes’.

• (iii)

It follows from (i) and (ii).

∎