Solving linear programs on factorized databases

TL;DR

This paper introduces a method to solve linear programming problems over database query answers by exploiting query structure, significantly reducing the problem size when the answer set is large, especially for queries with bounded fractional hypertree width.

Contribution

It presents a generic approach to rewrite linear programs based on factorized database representations, leveraging query structure to improve efficiency.

Findings

Rewrites linear programs to depend on factorized database size

Reduces linear program size for queries with bounded fractional hypertree width

Enables solving large LPs efficiently using query structure

Abstract

A typical workflow for solving a linear programming problem is to first write a linear program parametrized by the data in a language such as Math GNU Prog or AMPL then call the solver on this program while providing the data. When the data is extracted using a query on a database, this approach ignores the underlying structure of the answer set which may result in a blow-up of the size of the linear program if the answer set is big. In this paper, we study the problem of solving linear programming problems whose variables are the answers to a conjunctive query. We show that one can exploit the structure of the query to rewrite the linear program so that its size depends only on the size of the database and not on the size of the answer set. More precisely, we give a generic way of rewriting a linear program whose variables are the tuples in Q(D) for a conjunctive query Q and a database D into a linear program having a number of variables that only depends on the size of a factorized representation of Q(D), which can be much smaller when the fractional hypertree width of Q is bounded.

Figures

Equations

$$Q(p,r,d,f,l)=projects(p,f,l)\wedge researchers(r,f)\wedge developers(d,l).$$

$$\sum _{(p,r,d,f,l)\in Q}\omega (p,r,d,f,l)$$

$$\forall r\in D_{rname},\sum _{p,d,f,l}\omega (p,r,d,f,l)\le 100.$$

$$\forall d\in D_{dname},\sum _{p,r,f,l}\omega (p,r,d,f,l)\le 100.$$

$$\sum _{(p,f,l)\in pname}{\omega }_{pfl}^P$$

$$\forall f\in D_{field},\sum _{r\in rname}{\omega }_{rf}^R=\sum _{p\in pname}\sum _{l\in language}{\omega }_{pfl}^P$$

$$\forall l\in D_{language},\sum _{d\in dname}{\omega }_{dl}^D=\sum _{p\in pname}\sum _{f\in field}{\omega }_{pfl}^P$$

$$\forall r\in D_{rname},\sum _{f\in D_{field}}{\omega }_{rf}^R\le 100$$

$$\forall d\in D_{dname},\sum _{l\in D_{language}}{\omega }_{dl}^D\le 100$$

$${\mu }_1{X}_1+\cdots +{\mu }_n{X}_n.$$

$${\mu }_1{X}_1+\cdots +{\mu }_n{X}_n\le c.$$

$$\begin{array}{c}\text{max }\varphi \text{ subject to }\mathcal{C}\end{array}$$

$$\sum _{v\in pname}{S}_{p,v}$$

$$\forall v\in D_{rname},S_{r,v}\le 100$$

$$\forall v\in D_{dname},S_{d,v}\le 100$$

$$\sum _{d\in \mathsf{D}}{S}_{x,d}.$$

$$S_{x/d}=\underset{e\in \mathsf{Out}(u_{x/d})}{⨄}rel(C,e).$$

$$\underset{i\in \mathsf{In}(u)}{⨄}rel(C,i)=\underset{o\in \mathsf{Out}(u)}{⨄}rel(C,o).$$

$$\forall i\in \mathsf{In}(u):rel(C,i)=\underset{o\in \mathsf{Out}(u)}{⨄}rel(C,o)$$

$$\sum _{\begin{array}{c}\tau \in rel(C)\\ \tau (x)=d\end{array}}\omega (\tau )=\sum _{e\in \mathsf{Out}(x/d)}\sum _{\tau \in rel(C,e)}\omega (\tau )=\sum _{e\in \mathsf{Out}(x/d)}W(e)$$

$$\sum _{e\in \mathsf{Out}(u)}W(e)$$

$$\displaystyle=\sum_{\tau\in R}\omega(\tau)\text{\color[rgb]{.5,.5,.5} \quad(where $R=\biguplus_{e\in\mathsf{Out}(u)}\mathit{rel}(C,e)$)}$$

$$\sum _{\tau \in R}\omega (\tau )$$

$$\displaystyle=\sum_{e\in\mathsf{In}(u)}W(e)\text{\color[rgb]{.5,.5,.5} \quad(by definition of $W$)}$$

$$\sum _{e\in \mathsf{Out}(u)}W(e)$$

$$\displaystyle=W(i)\text{\color[rgb]{.5,.5,.5} \quad(by definition of $W$)}$$

$$W(e)=\sum _{\tau \in rel(C_u)}{\omega }_e(\tau ).$$

$$\sum _{\tau \in rel(C_u)}{\omega }_e(\tau )$$

$$=\sum _{\tau \in rel(C_{u_1})}\frac{W(e)}{W}{\omega }_{e_1}({\tau }_1)+\sum _{\tau \in rel(C_{u_2})}\frac{W(e)}{W}{\omega }_{e_2}({\tau }_2)$$

$$\sum _{\tau \in rel(C_u)}{\omega }_e(\tau )$$

$$=\frac{W(e)}{W}W(e_1)+\frac{W(e)}{W}W(e_2)$$

$$=W(e)\frac{W(e_1)+W(e_2)}{W}$$

$$=W(e).$$

$$\sum _{\tau \in rel(C_u)}{\omega }_e(\tau )$$

$$\sum _{\tau \in rel(C_u)}{\omega }_e(\tau )=W(e).$$

$${\omega }_e({\tau }^{\prime })=\sum _{\tau \in rel(C,e,{\tau }^{\prime })}\omega (\tau ).$$

Full text

1 Introduction

Computing the entire answer set of a database query is often only a first step toward a more complex processing of the data usually involving different aggregation tasks such as counting or computing average values. There has been a successful line of research to design algorithms that could aggregate the answers without explicitly solving the query first. For example, building on the celebrated result of Yannakakis [Yan81] and its extensions [GLS02], Pichler and Skritek [PS13] have shown that one can count the number of solutions of a quantifier-free acyclic (or of bounded hypertree width) conjunctive query in polynomial time in the size of the database, which can be significantly smaller than the size of the answer sets. In this paper, we are interested in an other kind of aggregation problems: solving optimization problems on the answers of a query. More precisely, we are interested in solving linear programs whose variables are the answers of a database query. This kind of problems is natural and it corresponds to the way most languages for optimization problems such as GNU MathProg and AMPL [FGK90] are designed. They allow to write an optimization problem by separating the definition of its constraints on parametrized sets – that can alternatively be seen as database tables – and a part describing the actual content of these sets – corresponding to the data stored in the table (see the example files of The AMPL Book [FGK93] for program examples). While these tools can easily be interfaced with a DBMS to access data, they do not receive any information concerning how the data was extracted from the database. In particular, when the data is the result of a query, it may be structured in a way that could be exploited by the solver to generate a smaller optimization problem. Before going further into details, we illustrate this phenomenon with a simple example showing how one can rewrite a linear program whose variables are the solutions to a conjunctive query into a smaller equivalent one.

1.1 Motivating example

To illustrate how one can exploit the structure of the query to improve the performances of the linear solver, we consider the following example. A university is funding several applied research projects where it is needed for researchers and developpers team up. Each project requires knowledge of a specific field and expertise in a programming language from the assigned researchers and developpers respectively. For simplicity, we assume here that each project needs exactly one field and one programming language. Moreover the researchers and developpers should work as pairs so for a given project the assigned researchers should work the same amount of time as the developpers. The goal of the university is to optimize the total time invested into the projects while ensuring that no contributor is overworked. The data about the researchers and projects is represented as follows:

• •

$projects(\underline{pname},field,language)$ lists the projects, the field they are related to and the required programming language,

• •

$researchers(\underline{rname},field)$ lists the researchers and the fields they are knowledgeable in,

• •

$developers(\underline{dname},language)$ lists the developpers and the programming languages they are proficient in.

All the possible assignments of contributors to projects can be expressed with a simple conjunctive query:

[TABLE]

Given a project $p$, language $l$, field $f$, researcher $r$ and developper $d$, we define $\omega(p,r,d,f,l)$ to be the percentage of time allotted for $d$ and $r$ to work on $p$, e.g. $\omega(p1,Alice,Eve,Databases,Python)=42\%$ means that $Alice$ and $Eve$ will work on $p1$ as a pair for $42\%$ of their time because of their respective skills in Databases and Python. The problem of maximizing the overall time spent on the projects without overloaded the contributors can be modelled as shown in Fig. 1.

Observe that this linear program model is independent of the data stored in the database which corresponds to the typical way of modelling linear programs using languages like AMPL or GNU MathProg. However, given a database $\mathbb{D}$, this model can be unfolded into a linear program whose size is proportional to the size of $Q(\mathbb{D})$. In this case, as $Q$ is the natural join of three tables, $Q(\mathbb{D})$ may be of size $O(N^{3})$ if we consider input relations of size bounded by $N$. Thus, it will results in a linear program having $O(N^{3})$ variables and $|D_{rname}|+|D_{dname}|$ constraints.

It turns out that the structure of the query, which is acyclic, allows us to rewrite the linear program into an equivalent one which has the same optimal value but has only $O(N)$ variables and a comparable number of constraints. The goal of this paper is to provide a generic way of doing this transformation when the query is sufficiently well-structured.

Observe that one could construct a similar example with any $k$ different types of skills, all being stored in $k$ different tables. The join query would now be between $k$ tables and it would give a naive linear program with $O(N^{k})$ variables. Our technique would still construct an equivalent linear program having $O(N)$ variables, leading to more interesting improvements.

We now illustrate how one can rewrite the original linear program into a smaller one. We introduce the following variables instead of the assignment weights. We denote by $\omega^{P}_{pfl}$ the time invested into the project $p$ by pairs of researchers and programmers in field $f$ and language $l$. We denote by $\omega^{R}_{rf}$ the percentage of time invested by researcher $r$ into projects in the field $f$. Similarly we define by $\omega^{D}_{dl}$ the amount of time invested by developer $d$ into projects requiring the language $l$.

Observe that, given $f$ and $l$, $\omega^{P}_{pfl}$ corresponds to $\sum\limits_{r,d}\omega^{P}_{prdfl}$ in the previous model. The objective function of the LP can thus be simplified through a change of variables. Given a field $f$, we want to enforce the total time invested into all the projects concerning field $f$ to be equal to the total time invested by researchers into projects in field $f$ which can be formally expressed as $\forall f\in D_{field},\sum\limits_{r\in rname}\omega^{R}_{rf}=\sum\limits_{p\in pname}\sum\limits_{l\in language}\omega^{P}_{pfl}$. Similarly we will express a constraint on the projects in a given language $l$ and developers working with language $l$. The ”pairing” constraint requiring that projects see the same amount of work from researchers and developers is implicitly enforced by the transitivity of the equality. Finally the constraints imposing an upper bound on the workload of the researchers and developers can easily be translated with these new weights. The complete rewriting of the linear program is shown in Fig. 2.

Observe that we introduce a single variable for each tuple of the input tables and that each of the four constraints above can be unfolded in a number of inequalities that is bounded by the cardinality of the domains of the attributes of the input tables. Thus, exploiting the acyclicity of $Q$, we transformed a linear program having $O(N^{3})$ variables to a linear program having $O(N)$ variables and the same optimal value. Our main result is an algorithm to automatically provide similar rewrittings for a fragment of linear programs on tables.

1.2 Contribution of this paper

In this paper, we generalize the technique seen in the previous example to reduce the size of a linear program whose variables are the answer set of a conjunctive query. More precisely, we give a generic approach to exploit the structure of conjunctive queries in order to generate equivalent linear programs of size polynomial in the size of the database which may be smaller than the size of the answer set. As deciding whether a conjunctive query has at least one solution is already $\mathsf{NP}$-complete, it is very unlikely that this approach is always possible. Hence, we based our approach on the framework of factorized databases of Olteanu and Závodný [OZ12]. A factorized database is a data structure that allows to represent a table in a factorized way that can be much smaller than the original table. More precisely, a factorized representation of a table may be seen as a circuit whose inputs are tables with one column and performing only Cartesian products and disjoint unions that we will refer to as $\{\uplus,\times\}$-circuits in this paper. Our central theorem, Theorem 4.6, shows that a given weighting of a table represented by a $\{\uplus,\times\}$-circuit can be canonically represented as a weighting of the edges of the circuit. This connection gives us a direct way of rewriting a linear program whose variables are the tuples of a table represented by a $\{\uplus,\times\}$-circuit into a linear program whose variables are the edges of the circuit having the same optimal value.

Several classes of conjunctive queries are known to admit polynomial size $\{\uplus,\times\}$-circuits, the most notorious one being conjunctive queries with bounded fractional hypertree width [OZ15]. Fractional hypertree width [GGLS16, GLS02] is a parameter generalizing acyclicity of the hypergraph of a conjunctive query [Yan81]. Intuitively, it measures how the relations and the variables of a conjunctive query interact with each other. Various aggregation problems are tractable on such queries such as counting the number of answers [PS13] or enumerating them with small delay [DK19, BDG07]. Most of these problems are solved by dynamic programming algorithms working in a bottom-up fashion on a tree shaped decomposition of the query. While one could use this dynamic programming approach to solve our problem as well, it makes the algorithm harder to describe. We thus choose to present our results on $\{\uplus,\times\}$-circuits directly. In this setting, given a $\{\uplus,\times\}$-circuit and a linear program on its answer set, the rewriting of the linear program is straightforward, even if the correction of the algorithm is more complicated to establish.

1.3 Related work

Factorized databases have already been used to solve complex problems on the answers of conjunctive queries. They have been used by Schleich et al. [SOC16] to perform linear regression on data represented as a factorized databases more efficiently than by first enumerating the complete answer set. Fink et al. [FHO12] have also used factorized databases to compute the probability distribution of query results in the framework of probabilistic databases. To the best of our knowledge, our result is the first to use this framework to solve optimization problems more efficiently.

Better integration of optimization problems directly into DBMS has however already been considered. Cadoli and Mancini [CM07] introduced an extension of SQL called ConSQL that allows to select optimal solutions to constraint problems directly in SQL-like queries. Šikšnys and Pedersen introduced SolveDB in [ŠP16] which is also an extension of SQL that allows to solve optimization problems directly in the querying language thus simplifying the usual workflow of feeding an AMPL programs with data extracted from a database. Both work mainly focus on designing the language and procedures to answer query in a very general setting. Though this is a very important and interesting question, we do not focus much on the language part in this work but more on the optimization part of solving linear programs.

The need to solve linear programs on answer sets of conjunctive queries has already appeared in the data mining community, especially for the problem of estimating the frequency of a pattern in a data graph. In this setting, the problem is to evaluate the frequency of a subgraph, the pattern, in a larger graph. A naive way of evaluating this frequency is to use the number of occurrences of the pattern as a frequency measure. Using this value as a frequency measure, is problematic since different occurrences of a pattern may overlap, and as such they share some kind of dependencies that is relevant from a statistical point of view. More importantly, due to the overlaps, this measure fails to be anti-monotone, meaning that a subpattern may be counter-intuitively matched less frequently than the pattern itself. Therefore, the finding of better anti-monotonic frequency measures – also known as support measures – has received a lot of attention in the data mining community [BN08, CRVD11, FB07]. A first idea is to count the maximal number of non-overlapping patterns [VGS02]. However, finding such a maximal subset of patterns essentially boils down to finding a maximal independent set in a graph, a notorious NP-complete problem [GJ02]. The $s$-measure, an alternative approach proposed by Wang et al. [WRF13], consists in counting the number of occurrences with respect to a weighting of the occurrences taking dependencies into account. The weighting proposed by the authors is defined via a linear program whose variables are the occurrences of the pattern. Seeing the problem of finding all occurrences of a pattern in a graph as a conjunctive query on a database whose only relation is the edges of the graph, our result directly applies to the problem of computing the $s$-measure and allows to compute it in polynomial time if the treewidth of the pattern is bounded.

Binary and Mixed-Integer programs (which are closely related to linear programs) have also appeared in the database community as ways to work with possible worlds. In [KPT13] the authors use binary programming to answer queries on inconsistent databases. In this work a binary programs concisely encodes the possible repairs of the database and is used to eliminate potential answers until a consistent answer (or the absence thereof) is found. In [MS12] the authors use Mixed-Integer Programming to answer how-to queries expressed in the ”Tiresias Query Language” (TiQL) which is based on datalog. There a Mixed-Integer Program is generated from the TiQL query and database. The MIP is then optimized using the structure of the database before being fed to an external solver which is similar in spirit to the goal of this paper.

Organization of the paper.

The paper is organized as follows. Section 2 introduces the notions and definitions that are necessary to precisely state our main result. In particular, this section contains the definition of the fragments of linear programs on tables we use in this paper. Section 3 presents our main technical tool to solve the general problem by relating weights given on the rows of a table and on the edges of the circuit representing it in a factorized manner and explains how a linear program whose variables are the elements of a relation defined by a circuit can be rewritten as a linear program whose variables are the edges of the circuit and thus be solved in polynomial time in the size of the circuit. The correction of our algorithm is presented in Section 4. Section 5 exposes some consequences of our results: we mention corollaries to prove the tractability of bounded fractional hypertreewidth quantifier-free conjunctive queries. We also present a simple method to handle existential quantification. Finally, we give suggestions for future work in the conclusion.

2 Preliminaries

2.1 General notations

We denote by $\mathbb{R}$ the set of real numbers and by $\mathbb{R}_{+}$ the set of non-negative real numbers. For any positive natural number $n$, we denote by $[n]$ the set $\{1,\dots,n\}$. For two sets $Y$ and $Z$, we denote by $Y^{Z}$ the set of (total) functions from $Z$ to $Y$. A finite function $R=\{(z_{1},y_{1}),\ldots,(z_{n},y_{n})\}$ where all $z_{i}$ are pairwise distinct is denoted by $[z_{1}/y_{1},\ldots,z_{n}/y_{n}]$. In the case $n=1$ we will simply write $z_{1}/y_{1}$ instead of $[z_{1}/y_{1}]$. Given a function $f\in Y^{Z}$ and $Z^{\prime}\subseteq Z$, we denote by $f_{\mid{Z^{\prime}}}$ the projection of $f$ on $Z^{\prime}$, that is, $f_{\mid{Z^{\prime}}}\in Y^{Z^{\prime}}$ and for every $z\in Z^{\prime}$, $f_{\mid{Z^{\prime}}}(z)=f(z)$

A directed graph is a pair $G=(V,E)$ with $E\subseteq V\times V$. Given a directed graph $G$ we write $\mathit{Nodes(G)}=V$ for the set of nodes and $\mathit{Edges(G)}=E$ for the set of edges. For any $u\in V$, we denote by $\mathsf{In}(u)$ (resp. $\mathsf{Out}(u)$) the set of ingoing (resp. outgoing) edges of $u$.

Let $\mathsf{X}$ be a finite set of attributes and $\mathsf{D}$ be a finite domain. A (database) tuple with attributes in $\mathsf{X}$ and domain $\mathsf{D}$ is a function $\tau:\mathsf{X}\to\mathsf{D}$ mapping attributes to database elements. A (database) relation with attributes $\mathsf{X}$ and domain $\mathsf{D}$ is a set of such tuples, that is, a subset of $\mathsf{D}^{\mathsf{X}}$.

2.2 Linear programs

Linear programs form a wide class of (convex) optimization problems which are known to be tractable. The first polynomial time algorithm for solving this problem is the ellipsoid algorithm [Kha79]. Other more efficient algorithms have been proposed later [Kar84, LS15] and there exist many tools using these techniques together with heuristics to solve this problem in practice such as lp_solve111http://lpsolve.sourceforge.net/ or glpk222https://www.gnu.org/software/glpk/. In this paper, we will use these tools as black boxes and will only use the fact that finding optimal solutions of linear programs can be done in polynomial time.

Given a finite set of variables $X$, a linear form $\phi$ on $X$ is given as a set of reals $\mu(x)$ for every $x\in X$. Given an assignment $a:X\rightarrow\mathbb{R}$ of the variables of $X$, $\phi(a)$ is defined as $\sum_{x\in X}\mu(x)a(x)$. Through the paper, we denote linear forms as polynomials as follows:

[TABLE]

A linear constraint $C=(\phi,s,c)$ on variables $X$ is given as a linear form $\phi$ on $X$, a binary relation $s\in\{\leq,=,\geq,<,>\}$ and a real $c\in\mathbb{R}$. Given an assignment $a:X\rightarrow\mathbb{R}$ of the variables of $X$, $C$ is satisfied by $a$ if $s(\phi(a),c)$. Through the paper, we denote linear constraints by adapting the notation of linear programs:

[TABLE]

A linear program $L=(X,\phi,\mathcal{C})$ on variables $X$ is given as a linear form $\phi$ called the objective function and a set of linear constraints $\mathcal{C}$ on variables $X$. We write $L$ as:

[TABLE]

The feasible region of $L$, denoted by $sol(L)$, is the set of assignments $a:X\rightarrow\mathbb{R}$ such that for every $C\in\mathcal{C}$, $a$ satisfies $C$. We will often refer to an assignment in the feasible region as a solution to the linear program. An optimal solution of $L$ is a solution $a$ in the feasible region that maximizes the objective function of $L$, i.e. for all $a^{\prime}\in sol(L)$, $\phi(a)\geq\phi(a^{\prime})$. The optimal value of $L$ is defined as $\phi(a)$ where $a$ is some optimal solution of $L$.

The size of $L$ is defined to be $|X|\cdot|\mathcal{C}|$. Observe that the size of an encoding of a linear program may be bigger than our definition of size as we will have to store the coefficient. In this paper, we present a succinct rewriting of a class of linear program that does not change the coefficients, so we do not need to be too precise on the way they are encoded. In order to simplify the presentation of the results, we will not take this encoding into account.

2.3 Linear programs on relations

In this paper, we are interested in a class of linear programs whose variables are the elements of a finite relation. To manipulate this kind of programs, we need a definition of linear programs parametrized by a relation. We introduce in this section the notion of Common Attribute Sum Linear Programs, abbreviated as CAS-LPs. A CAS-LP $L=(\mathsf{X},\mathsf{D},\phi,\mathcal{C})$ on attributes $\mathsf{X}$ and domain $\mathsf{D}$ is given as a linear form $\phi$ called the objective function and a set of linear constraints, both on variables $S(\mathsf{X},\mathsf{D}):=\{S_{x,d}\}_{x\in\mathsf{X},d\in\mathsf{D}}$, called the CAS variables.

From a strictly formal point of view, a CAS-LP is just a linear program on variables $S(\mathsf{X},\mathsf{D})$. However, we are not interested in the optimal solutions of this linear program but in the solution of a linear program that is obtained from a CAS-LP and a relation $R\subseteq\mathsf{D}^{\mathsf{X}}$.

Given a CAS-LP $L$ on attributes $\mathsf{X}$, domain $\mathsf{D}$ and given a relation $R\subseteq\mathsf{D}^{\mathsf{X}}$, a solution of $L$ on $R$ is an assignment $\omega:R\rightarrow\mathbb{R}_{+}$ such that the extension of $\omega$ on variables $S_{x,d}$ defined as $\omega(S_{x,d}):=\sum\limits_{\begin{subarray}{c}\tau\in R \\ \tau(x)=d\end{subarray}}\omega(\tau)$ is a solution of $L$. An optimal solution of $L$ on $R$ is a solution of $L$ on $R$ that maximizes the objective function of $L$. The optimal value of $L$ on $R$ is the value of the objective function on an optimal solution.

Alternatively, we could see a solution of $L$ on $R$ as the solution of the ground linear programs $L(R)$ on variables $\{\tau\mid\tau\in R\}$ obtained by replacing the variable $S_{x,d}$ by $\sum\limits_{\begin{subarray}{c}\tau\in R \\ \tau(x)=d\end{subarray}}\tau$ in $L$ and by adding the constraints $\tau\geq 0$ for every $\tau\in R$. This gives a first naive algorithm to find the optimal solution of $L$ on $R$: one can compute $L(R)$ explicitly and solve it using an external solver.

When $R$ is given as the relation defined by a conjunctive query on a database however, the relation may be much bigger than the size of the database which results in a huge ground linear program. The goal of this paper is to study how one can compute the optimal value of such ground linear programs in time polynomial in the size of the database and not in the size of the resulting relation.

Fig. 3 gives a CAS-LP $L$ corresponding to the example of Section 1.1. Indeed, if $R$ is the relation corresponding to the answer set of query $Q$ of Section 1.1, then $L(R)$ exactly corresponds to the linear program of Fig. 1.

Actually, computing this optimal value for conjunctive query is $\mathsf{NP}$-hard:

Proposition 2.1**.**

The problem of deciding whether the optimal value of $L(Q(\mathbb{D}))$ is non-zero given a quantifier-free conjunctive query $Q$, a database $\mathbb{D}$ and a CAS-LP $L$ is $\mathsf{NP}$-hard.

Proof.

Let $Q$ be a quantifier-free conjunctive query and $x$ an attribute of $Q$ on domain $\mathsf{D}$. Let $L$ be the CAS-LP defined as:

[TABLE]

It is not hard to see that, given a database $\mathbb{D}$, the optimal value of $L(Q(\mathbb{D}))$ is non-zero if and only if $Q(\mathbb{D})\neq\emptyset$. Since deciding, given $Q$ and $\mathbb{D}$, whether $Q(\mathbb{D})\neq\emptyset$ is an $\mathsf{NP}$-complete problem [CM77], the statement of the proposition follows. ∎

2.4 $\{\uplus,\times\}$-Circuits

To solve ground linear programs efficiently, we will work on factorized representations of relations. We use a generalization of the framework of factorized databases introduced by Olteanu et al. [OZ12]. Our factorized representations are defined similarly with only a subset of the syntactic restrictions (we do not need $f$-representation or $d$-representations) on the circuits since our algorithm does not need all of them to work.

A $\{\cup,\times\}$-circuit on attributes $\mathsf{X}$ and domain $\mathsf{D}$ is a directed acyclic graph $C$ with edges directed from the leaves to a single root $r$ called the output of $C$ and whose nodes, called gates, are labeled as follows:

• •

internal gates are labeled by either $\times$ or $\cup$,

• •

inputs, i.e. nodes of in-degree [math], are labeled by atomic database relations of the form $x/d$ for an attribute $x$ and an element $d\in\mathsf{D}$. 333Observe that every possible input relation could be easily simulated with this restricted input using $\times$ and $\cup$. For example, the relation $\{[x/1,y/1,z/1],[x/0,y/0,z/0]\}$ can be rewritten as $(x/1\times y/1\times z/1)\cup(x/0\times y/0\times z/0)$.

Figure 4 pictures a $\{\cup,\times\}$-circuit on attributes $\{x,y,z\}$ and domain $\{0,1\}$.

We denote by $\mathsf{Attr}(C)$ the attributes appearing in the database relations of the input nodes of $C$. We define the size of $C$ as the number of edges in the underlying graph of $C$. We denote it by $|C|$.

Given a gate $u$ of $C$, we denote by $C_{u}$ the subcircuit of $C$ rooted in $u$ (in which dangling edges from above are removed). We denote by $\times$-gates$(C)$ the set of all the $\times$-gates and by $\cup$-gates$(C)$ the set of all $\cup$-gates of $C$.

We impose that any $\{\cup,\times\}$-circuit must satisfy the following restrictions.

• •

If $u$ is a $\cup$-gate with children $u_{1},\dots,u_{k}$, we impose that $\mathsf{Attr}(C_{u_{1}})=\dots=\mathsf{Attr}(C_{u_{k}})$.

• •

If $u$ is a $\times$-gate with children $u_{1},\dots,u_{k}$, we impose that $\mathsf{Attr}(C_{u_{i}})\cap\mathsf{Attr}(C_{u_{j}})=\emptyset$ for every $i<j\leq k$. Observe that it implies that $C_{u_{i}}$ and $C_{u_{j}}$ are disjoint graphs since otherwise they would share an input and, thus, an attribute of the relation of this input node.

Due to these restrictions, any gate $u$ of a $\{\cup,\times\}$-circuit $C$ specifies a database relation $\mathit{rel}(C_{u})$ that we define inductively as follows. If $u$ is an input then $\mathit{rel}(C_{u})$ is the relation labelling $u$. If $u$ is a $\times$-gate with children $u_{1},\dots,u_{k}$, $\mathit{rel}(C_{{u}})$ is defined as $\mathit{rel}(C_{u_{1}})\times\dots\times\mathit{rel}(C_{u_{k}})$. If $u$ is a $\cup$-gate with children $u_{1},\dots,u_{k}$, $\mathit{rel}(C_{{u}})$ is defined as $\mathit{rel}(C_{u_{1}})\cup\dots\cup\mathit{rel}(C_{u_{k}})$. We define $\mathit{rel}(C)=\mathit{rel}(C_{r})$ where $r$ is the output of the circuit.

The way we have currently defined circuits is not yet enough to allow for efficient aggregation. Given a $\{\cup,\times\}$-circuit $C$ and an $\cup$-gate $u$ of $C$ with children $u_{1},\dots,u_{k}$, we say that $u$ is disjoint if $\mathit{rel}(C_{{u_{i}}})\cap\mathit{rel}(C_{{u_{j}}})=\emptyset$ for every $i<j\leq k$. In our figures and proofs, we will use the symbol $\uplus$ to indicate that a union is disjoint. A $\{\uplus,\times\}$-circuit is a $\{\cup,\times\}$-circuit in which every $\cup$-gate is a $\uplus$-gate.

Observe that given a $\{\uplus,\times\}$-circuit $C$, one can compute the cardinality of its relation $|\mathit{rel}(C)|$ in time linear in the size of $C$ by a simple inductive algorithm that adds the sizes of the children of a $\uplus$-gate and multiply the sizes of the children of the $\times$-gates.

Disjointness is a semantic condition and it is $\mathsf{coNP}$-hard to check the disjointness of a gate given a $\{\cup,\times\}$-circuit. In practice however, algorithms producing $\{\uplus,\times\}$-circuit ensure the disjointess of all $\cup$-gates during the construction of the circuit, usually by using $\cup$-gates of the form: $\biguplus_{d\in D}(\{x/d\}\times C_{d})$, so that the different values $d$ assigned to $x$ ensure disjointness.

Theorem 2.2**.**

[OZ15, Theorem 7.1]** Given a quantifier free conjunctive query $Q$, a hypertree decomposition $T$ of $Q$ of fractional hypertreewidth $k$ and a database $\mathbb{D}$, one can construct a $\{\uplus,\times\}$-circuit $C$ of size at most $|Q|\cdot|\mathbb{D}|^{k}$ whose attributes are the variables of $Q$ such that $Q(\mathbb{D})=\mathit{rel}(C)$ in time $O(\log(|\mathbb{D}|)\cdot|\mathbb{D}|^{k})$.

For the special case of quantifier free acyclic conjunctive query the theorem follows from a variant of Yanakakis’ algorithm. We refer the reader to [GGLS16] for the definition of fractional hypertreewidth as we will not work with this notion directly in this paper but only with $\{\uplus,\times\}$-circuits, using Theorem 2.2 to bridge both notions.

3 Rewriting linear programs

The goal of this paper is to show that one can rewrite the ground linear program $L(R)$ of a CAS-LP $L$ when the relation $R$ is structured. More precisely, we show the following theorem:

Theorem 3.1**.**

Given a CAS-LP $L$ with $m$ constraints and a $\{\uplus,\times\}$-circuit $C$ both on attributes $X$ and domain $D$, one can construct in polynomial time a linear program $L^{\prime}$ having $|C|$ variables and $m+O(|C|)$ constraints such that $L^{\prime}$ and $L(\mathit{rel}(C))$ have the same optimal value.

3.1 Construction of the smaller linear program

We start by presenting the construction of the new linear program of Theorem 3.1. We fix $L$ a CAS-LP with $m$ constraints and $C$ a $\{\uplus,\times\}$-circuit, both on attributes $X$ and domain $D$. We construct a linear program $L^{\prime}$ having the same optimal value as $L(\mathit{rel}(C))$. The variables of $L^{\prime}$ are the edges of $C$. $L^{\prime}$ has a first set of constraints that we call the soundness constraints that are the following. For every edge $e$ of $C$, we have a constraint $e\geq 0$ and for every gate $u$ of $C$,

• •

if $u$ is a $\uplus$-gate that is not the output of the circuit then we have the constraint : $\sum_{i\in\mathsf{In}(u)}i=\sum_{o\in\mathsf{Out}(u)}o$,

• •

if $u$ is a $\times$-gate then let $i_{1},\dots,i_{k}$ be the ingoing edges of $u$ in an arbitrary order. We have the constraint $i_{p}=i_{p+1}$ for $1\leq p<k$. If $u$ is not the output of the circuit, we also have the constraint $i_{1}=\sum_{o\in\mathsf{Out}(u)}o$.

Observe that there are at most $3|C|$ soundness constraints in $L^{\prime}$. Now, we add a constraint $c^{\prime}$ for each constraint $c$ of $L$ that is obtained as follows: we replace every occurrence of $S_{x,d}$ in $c$ by $\sum_{e\in\mathsf{Out}(x/d)}e$ for every $x\in X$ and $d\in D$ where $\mathsf{Out}(x/d)$ denotes the set of edges of $C$ going out of an input labeled by $x/d$. The objective function of $L^{\prime}$ is obtained by applying the same replacement of $S_{x,d}$ variables in the objective function of $L$.

The number of variables of $L^{\prime}$ is $|C|$ and that it has $m+3|C|$ constraints so it has the required size. It is also clear that one can easily construct $L^{\prime}$ from $L$ and $C$ in polynomial time: it is indeed sufficient to visit each gate of the circuit to generate the soundness constraints and then to replace each occurrences of $S_{x,d}$ by the appropriate sum.

It turns out that the optimal values of $L^{\prime}$ and of $L(\mathit{rel}(C))$ coincide. To prove this however, we have to understand the structure of $\{\uplus,\times\}$-circuit better and the rest of the paper is dedicated to prove the correction of our construction.

3.2 Correctness of the construction

A solution of $L(\mathit{rel}(C))$ can be seen as a weighting of the tuples $\tau\in\mathit{rel}(C)$. Similarly, a solution of $L^{\prime}$ can be seen as a weighting of the edges of $C$. We will show strong connections between both weightings that will allow us to prove that $L(\mathit{rel}(C))$ and $L^{\prime}$ have the same optimal value.

Let $C$ be a $\{\uplus,\times\}$-circuit. A tuple-weighting $\omega$ of $C$ is defined as a variable assignment of $\mathit{rel}(C)$, e.g. $\omega\in\mathbb{R}_{+}^{\mathit{rel}(C)}$. An edge-weighting $W$ of $C$ is similarly defined as a variable assignment of $\mathit{Edges(C)}$, e.g. $W\in\mathbb{R}_{+}^{\mathit{Edges(C)}}$. Throughout the paper, we will always use the symbol $\omega$ for tuple-weightings and $W$ for edge-weightings.

Definition 3.2**.**

An edge-weighting $W$ is sound if for every gate $u$ of $C$ we have:

• •

if $u$ is a $\uplus$-gate that is not the output of the circuit then $\sum_{i\in\mathsf{In}(u)}W(i)=\sum_{o\in\mathsf{Out}(u)}W(o)$,

• •

if $u$ is a $\times$-gate then $\forall i,i^{\prime}\in\mathsf{In}(u)$: $W(i)=W(i^{\prime})$. If $u$ is not the output of the circuit, we also have $W(i)=\sum_{o\in\mathsf{Out}(u)}W(o)$.

Given a tuple-weighting $\omega$ and a tuple-weighting $W$ we say that $W$ is compatible with $\omega$ iff $\forall x\in X,\forall d\in\mathsf{D}\sum\limits_{\begin{subarray}{c}\tau\in\mathit{rel}(C) \\ \tau(x)=d\end{subarray}}\omega(\tau)=\sum\limits_{e\in\mathsf{Out}(x/d)}W(e)$.

The correctness of our construction directly follows from the next theorem whose proof is delayed to Section 4, connecting tuple-weightings with sound edge-weightings.

Theorem 3.3**.**

Given a $\{\uplus,\times\}$-circuit $C$.

1. (a)

For every tuple-weighting $\omega$ of $C$, there exists a sound edge-weighting $W$ such that $W$ is compatible with $\omega$. 2. (b)

Given a sound edge-weighting $W$ of $C$, there exists a tuple-weighting $\omega$ of $C$ such that $W$ is compatible with $\omega$.

We now explain how one can use Theorem 3.3 to prove the correctness of our transformation.

Let $\omega:\mathit{rel}(C)\rightarrow\mathbb{R}_{+}$ be a tuple-weighting that is a solution of $L(\mathit{rel}(C))$. Let $W:\mathit{Edges(C)}\rightarrow\mathbb{R}_{+}$ be the sound edge-weighting compatible with $\omega$ that is given by Theorem 3.3. We claim that $W$ is a solution of $L^{\prime}$. First, $W$ satisfies all the soundness constraints of $L^{\prime}$ since $W$ is sound. Moreover, since $W$ is compatible with $\tau$, $\forall x\in X,\forall d\in\mathsf{D}\sum\limits_{\begin{subarray}{c}\tau\in\mathit{rel}(C) \\ \tau(x)=d\end{subarray}}\omega(\tau)=\sum\limits_{e\in\mathsf{Out}(x/d)}W(e)$. But this is the value of $\omega(S_{x,d})$, which means that $W$ is a solution of $L^{\prime}$. It tells us that the optimal value of $L^{\prime}$ is bigger than the optimal value of $L$.

Now let $W:\mathit{Edges(C)}\rightarrow\mathbb{R}_{+}$ be an edge-weighting of $C$ that is a solution of $L^{\prime}$ and let $\omega$ be a tuple-weighting compatible with $W$. We show that $\omega$ is a solution of $L(\mathit{rel}(C))$. Indeed, by definition, $L^{\prime}$ is obtained by replacing $S_{x,d}$ with $\sum\limits_{e\in\mathsf{Out}(x/d)}e$. Since $\omega$ is compatible with $W$, we also have $\forall x\in X,\forall d\in\mathsf{D}\sum\limits_{\begin{subarray}{c}\tau\in\mathit{rel}(C) \\ \tau(x)=d\end{subarray}}\omega(\tau)=\sum\limits_{e\in\mathsf{Out}(x/d)}W(e)$. But this is $\omega(S_{x,d})$ by definition, which means that $\omega$ is a solution of $L(\mathit{rel}(C))$. Thus the optimal value of $L(\mathit{rel}(C))$ is bigger than the optimal value of $L^{\prime}$. Hence, $L(\mathit{rel}(C))$ and $L^{\prime}$ have the same optimal value.

The rest of this paper is dedicated to the proof of Theorem 3.3.

4 Connecting edge-weightings and tuple-weightings

In this section, we prove Theorem 3.3. Actually, we prove a stronger version of this theorem presented in Section 4.2 that gives a better insight into how tuple-weighting and edge-weighting relate to each others. This stronger version allows us to prove the result by induction on the circuit. Before stating the theorem, we however need to introduce some notions on circuits.

4.1 Proof-trees

From now on, to simplify the proofs, we assume wlog that every internal gate of a $\{\uplus,\times\}$-circuit has fan-in two. It is easy to see that, by associativity, $\times$-gates and $\uplus$-gates of fan-in $k>2$ can be rewritten with $k-1$ similar gates which is a polynomial size transformation of the circuit. We also assume that for every $x\in\mathsf{Attr}(C)$ and $d\in D$, we have at most one input labeled with $x/d$. This can be easily achieved by merging all such inputs.

Let $C$ be a $\{\uplus,\times\}$-circuit and let $\tau\in \mathit{rel}(C)$. The proof-tree of $\tau$, denoted $\mathsf{T}_{C}(\tau)$, is a subcircuit of $C$ participating to the computation of $\tau$. More formally, $\mathsf{T}_{C}(\tau)$ is defined inductively by starting from the output as follows: the output of $C$ is in $\mathsf{T}_{C}(\tau)$. Now if $u$ is a gate in $\mathsf{T}_{C}(\tau)$ and $v$ is a child of $u$, then we add $v$ in $\mathsf{T}_{C}(\tau)$ if and only if $\tau_{\mid\mathsf{Attr}(C_{v})}\in\mathit{rel}(C_{v})$. A proof-tree is depicted in red in Figure 5.

Proposition 4.1**.**

Given a $\{\uplus,\times\}$-circuit $C$ and $\tau\in\mathit{rel}(C)$, $\mathsf{T}:=\mathsf{T}_{C}(\tau)$, the following holds:

• •

every $\times$-gate of $\mathsf{T}$ has all its children in $\mathsf{T}$,

• •

every $\uplus$-gate of $\mathsf{T}$ has exactly one of its children in $\mathsf{T}$,

• •

$\mathsf{T}$* is connected and every gate of $\mathsf{T}$ has out-degree at most $1$ in $\mathsf{T}$,*

• •

for any $x\in\mathsf{Attr}(C)$, $\mathsf{T}$ contains exactly one input labeled with $x/\tau(x)$.

Proof.

By induction, it is clear that for every gate $u$ of $\mathsf{T}$, $\tau_{\mid\mathsf{Attr}(C_{u})}\in\mathit{rel}(C_{u})$. Thus, if $u$ is a $\uplus$-gate of $\mathsf{T}$, as $u$ is disjoint, exactly one of it child $v$ has $\tau_{\mid\mathsf{Attr}(C_{u})}\in\mathit{rel}(C_{u})$. If $u$ is a $\times$-gate with children $u_{1},u_{2}$, then by definition, $\tau_{\mid\mathsf{Attr}(C_{u})}\in\mathit{rel}(C_{u})$ if and only if $\tau_{\mid\mathsf{Attr}(C_{u_{1}})}\in\mathit{rel}(C_{u_{1}})$ and $\tau_{\mid\mathsf{Attr}(C_{u_{2}})}\in\mathit{rel}(C_{u_{2}})$. Thus both $u_{1}$ and $u_{2}$ are in $\mathsf{T}$.

It is clear from definition that $\mathsf{T}$ is connected since $\mathsf{T}$ is constructed by inductively adding children of gates that are already in $\mathsf{T}$. Now assume that $\mathsf{T}$ has a gate $u$ of out-degree greater than $1$ in $\mathsf{T}$. Let $u_{1},u_{2}$ be two of its parents and let $v$ be their least common ancestor in $\mathsf{T}$. By definition, $v$ has in-degree $2$, this it is necessarily a $\times$-gates with children $v_{1},v_{2}$. Thus $u$ is both in $C_{v_{1}}$ and $C_{v_{2}}$, which is impossible since they are disjoint subcircuits.

Finally, let $x\in\mathsf{Attr}(C)$. Observe that if $\mathsf{T}$ has an input labeled $u$ with $x/d$ then $d=\tau(x)$ since $\tau_{\mid\mathsf{Attr}(C_{u})}\in\mathit{rel}(C_{u})$. Thus, if $\mathsf{T}$ contains two inputs $v_{1}$ and $v_{2}$ on attribute $x$, they are both labeled with $x/\tau(x)$ and are thus the same input. ∎

A proof-tree may be seen as the only witness that a tuple belongs to $\mathit{rel}(C)$. We define the relation induced by an edge $e$ as the set of tuples of $\mathit{rel}(C)$ such that their proof-tree contains the edge $e$, that is $\mathit{rel}(C,e):=\{\tau\in\mathit{rel}(C)\mid e\in\mathsf{T}_{C}(\tau)\}$.

In the following we show some fundamental properties of proof-trees that we will use to prove the correctness of our algorithm. For the rest of this section, we fix a $\{\uplus,\times\}$-circuit $C$ on attributes $\mathsf{X}$ and domain $\mathsf{D}$.

Proposition 4.2**.**

Let $u$ be a gate of $C$ and $e_{1},e_{2}$ $\in\mathsf{Out}(u)$ with $e_{1}\neq e_{2}$. Then $\mathit{rel}(C,e_{1})\cap\mathit{rel}(C,e_{2})=\emptyset$.

Proof.

Assume that $\tau\in\mathit{rel}(C,e_{1})$ and $\tau\in\mathit{rel}(C,e_{2})$ for $e_{1},e_{2}\in\mathsf{Out}(u)$ with $e_{1}\neq e_{2}$. By definition, it means that both $e_{1}$ and $e_{2}$ are in $\mathsf{T}_{C}(\tau)$ which is a contradiction as $\mathsf{T}_{C}(\tau)$ has out-degree $1$ by Proposition 4.1. ∎

Proposition 4.3**.**

Let $x\in\mathsf{X}$ and $d\in\mathsf{D}$ and let $u_{x/d}$ be the input of $C$ labeled with $x/d$.

[TABLE]

Proof.

Let $x\in\mathsf{Attr}(C)$ and $d\in\mathsf{D}$. Let $\tau\in S_{x/d}$. By Proposition 4.1, $u_{x/d}$ belongs to $\mathsf{T}_{C}(\tau)$ so there is at least one $e\in\mathsf{Out}(u_{x/d})$ that belongs to $\mathsf{T}_{C}(\tau)$. Similarly, if $\tau\in\mathit{rel}(C,e)$ for some $e\in\mathsf{Out}(u_{x/d})$ then $u_{x/d}$ is also in $\mathsf{T}_{C}(\tau)$, and then $\tau(x)=d$. This proves the equality. The disjointness of the union directly follows from Proposition 4.2. ∎

Next we show that the disjoint union of the relations induced by each ingoing edge of a $\uplus$-gate is equal to the disjoint union of the relations induced by each outgoing edge of this gate.

Proposition 4.4**.**

*Let $u$ be an internal $\uplus$-gate of $C$,

[TABLE]

Proof.

Let $S_{u}:=\{\tau\mid u\in\mathsf{T}_{C}(\tau)\}$ and $\tau\in S_{u}$. It is clear from Proposition 4.1 that a proof tree contains $u$ if and only if it contains at least an edge of $\mathsf{In}(u)$ and at least an edge of $\mathsf{Out}(u)$. Thus, both unions are equal to $S_{u}$.

The disjointness of the first union directly follows from the second item of Proposition 4.1 and the disjointness of the second union from Proposition 4.2. ∎

Finally we show that the disjoint union of the relations induced by the outgoing edges of a $\times$-gate is equal to the relation induced by each ingoing edge of this gate.

Proposition 4.5**.**

Let $u$ be an internal $\times$-gate of $C$,

[TABLE]

Proof.

Let $S_{u}:=\{\tau\mid u\in\mathsf{T}_{C}(\tau)\}$. By Proposition 4.1, it is clear that if $u$ is in a proof tree $\mathsf{T}$, then every edge of $\mathsf{In}(u)$ is also in $\mathsf{T}$ and exactly one edge of $\mathsf{Out}(u)$ is in $\mathsf{T}$. Thus, both sets in the statement are equal to $S_{u}$. The disjointness of the union follows directly from Proposition 4.2. ∎

4.2 Theorem statement

Given a tuple-weighting $\omega$ of a $\{\uplus,\times\}$-circuit $C$, it naturally induces an edge-weighting $W$ defined for every $e\in\mathit{Edges(C)}$ as $W(e):=\sum_{\tau\in\mathit{rel}(C,e)}\omega(e)$. We call $W$ the edge-weighting induced by $\omega$.

Theorem 4.6**.**

Given a $\{\uplus,\times\}$-circuit $C$.

1. (a)

For every tuple-weighting $\omega$ of $C$, the edge-weighting $W$ induced by $\omega$ is sound. 2. (b)

Given a sound edge-weighting $W$ of $C$, there exists a tuple-weighting $\omega$ of $C$ such that $W$ is the edge-weighting induced by $\omega$.

We will provide the proof for this theorem in the following sections.

Observe that if if an edge-weighting $W$ is induced by a tuple-weighting $\omega$ then

[TABLE]

so $W$ is also compatible with $\omega$.

Thus Theorem 4.6 is indeed a generalization of Theorem 3.3.

4.3 Proof of Theorem 4.6 a

This section is dedicated to the proof of Theorem 4.6 a, namely that, given a $\{\uplus,\times\}$-circuit $C$ and a tuple-weighting $\omega$ of $C$, the edge-weighting $W$ of $C$ induced by $\omega$ defined as $W(e):=\sum_{\tau\in\mathit{rel}(C,e)}\omega(\tau)$ is sound.

We will see that the soundness of $W$ follows naturally from the properties of proof-trees of the previous section. We prove that $W$ is sound by checking the case of $\uplus$-gates and $\times$-gates separately.

• •

We first have to show that for every $\uplus$-gate $u$ of $C$, it holds that $\sum_{e\in\mathsf{Out}(u)}W(e)=\sum_{e\in\mathsf{In}(u)}W(e)$. This is a consequence of Proposition 4.4. Let $u$ be a $\uplus$-gate of $C$.

[TABLE]

The disjointness of the union in $R$ has been proven in Proposition 4.4, which also states that $R=\biguplus_{e\in\mathsf{In}(u)}\mathit{rel}(C,e)$. Thus, the last term in the sum can be rewritten as

[TABLE]

• •

We now show that for every $\times$-gate $u$ of $C$ and for every edge $i\in\mathsf{In}(u)$ going in $u$, it holds that $\sum_{e\in\mathsf{Out}(u)}W(e)=W(i)$. Let $u$ be a $\times$-gate and $i\in\mathsf{In}(u)$. The proof is very similar to the previous case but is now a consequence of Proposition 4.5. As before, we have $\sum_{e\in\mathsf{Out}(u)}W(e)=\sum_{\tau\in R}\omega(\tau)$ where $R=\biguplus_{e\in\mathsf{Out}(u)}\mathit{rel}(C,e)$. The disjointness of the union in $R$ has been proven in Proposition 4.5, which also implies that $R=\mathit{rel}(C,i)$. Thus, we have:

[TABLE]

4.4 Proof of Theorem 4.6 b

This section is dedicated to the proof of Theorem 4.6 b, namely that, given a circuit $W$ and a sound edge-weighting $W$ of $C$, there exists a tuple-weighting $\omega$ of $C$ such that, for every edge $e$ of $C$, $\sum_{\tau\in\mathit{rel}(C,e)}\omega(\tau)=W(e)$.

In this section, we fix a $\{\uplus,\times\}$-circuit $C$ and a sound edge-weighting $W$ of its edges. We assume wlog that the root $r$ of $C$ has a single ingoing edge which we call the output edge $o_{r}$. We construct $\omega$ by induction. For every edge $e=\langle u,v\rangle$ of $C$, we inductively construct a tuple weighting $\omega_{e}$ of $\mathit{rel}(C_{u})$. We will then choose $\omega=\omega_{o_{r}}$ and show that this tuple-weighting verifies $\sum_{\tau\in\mathit{rel}(C,e)}\omega(\tau)=W(e)$ for every edge $e$ of $C$.

For $e=\langle u,v\rangle$, we define $\omega_{e}:\mathit{rel}(C_{u})\rightarrow\mathbb{R}_{+}$ inductively as follows:

• •

Case 1: $u$ is an input labeled with $x/d$ then $\mathit{rel}(C_{u})$ contains only the tuple $x/d$. We define $\omega_{e}(x/d):=W(e)$.

• •

Case 2: $u$ is a $\uplus$-gate with children $u_{1},u_{2}$. Let $e_{1}=\langle u_{1},u\rangle$ and $e_{2}=\langle u_{2},u\rangle$. In this case, given $\tau\in\mathit{rel}(C_{u})$, we have by definition that $\tau\in\mathit{rel}(C_{u_{1}})$ or $\tau\in\mathit{rel}(C_{u_{2}})$. Assume wlog that $\tau\in\mathit{rel}(C_{u_{1}})$. If $\sum_{f\in\mathsf{Out}(u)}W(f)\neq 0$, we define: $\omega_{e}(\tau):=W(e){\omega_{e_{1}}(\tau)\over W(e_{1})+W(e_{2})}$. Observe that since $W$ is sound, $W(e_{1})+W(e_{2})=\sum_{f\in\mathsf{Out}(u)}W(f)\neq 0$. Otherwise $\omega_{e}(\tau):=0$.

• •

Case 3: $u$ is a $\times$-gate with children $u_{1},u_{2}$. Let $e_{1}=\langle u_{1},u\rangle$ and $e_{2}=\langle u_{2},u\rangle$. In this case, given $\tau\in\mathit{rel}(C_{u})$, we have by definition that $\tau=\tau_{1}\times\tau_{2}$ with $\tau_{1}\in\mathit{rel}(C_{u_{1}})$ and $\tau_{2}\in\mathit{rel}(C_{u_{2}})$. If $W(e_{1})\neq 0$ and $W(e_{2})\neq 0$, we define: $\omega_{e}(\tau):=W(e){\omega_{e_{1}}(\tau_{1})\over W(e_{1})}{\omega_{e_{2}}(\tau_{2})\over W(e_{2})}$. Otherwise $\omega_{e}(\tau):=0$.

We begin our proof by showing that as we construct each $\omega_{e}$ through a bottom-up induction there is a relation between $W(e)$ and $\omega_{e}$ which resembles the property we aim to prove in this section.

Lemma 4.7**.**

For every gate $u$ of $C$ and $e=\langle u,v\rangle\in\mathit{Edges(C)}$,

[TABLE]

Proof.

Let $e=\langle u,v\rangle$ be an edge of $C$. Observe that when $\sum_{o\in\mathsf{Out}(u)}W(o)=0$, then $W(e)=0$ since $e\in\mathsf{Out}(u)$ and $W$ has positive value. Moreover, by definition of $\omega_{e}$, for every $\tau\in\mathit{rel}(C_{u})$, $\omega_{e}(\tau)=0$. In particular, $\sum_{\tau\in\mathit{rel}(C_{u})}\omega_{e}(\tau)=0=W(e)$. In this case then, the lemma holds.

In the rest of the proof, we now assume that $\sum_{o\in\mathsf{Out}(u)}W(o)\neq 0$. We show the lemma by induction on the nodes of $C$ from the leaves to the root.

Base case: $u$ is a leaf labelled with $x/d$. Let $e$ be an outgoing edge of $u$. Observe that $\mathit{rel}(C_{u})$ contains a single tuple $x/d$ and that, by definition of $\omega_{e}$, $W(e)=\omega_{e}(x/d)=\sum_{\tau\in\mathit{rel}(C_{u})}\omega_{e}(\tau)$.

Inductive case: Now let $u$ be an internal gate of $C$ with children $u_{1},u_{2}$ and let $e_{1}=\langle u_{1},u\rangle$ and $e_{2}=\langle u_{2},u\rangle$, as depicted in Figure 6.

Case 1: Assume that $u$ is a $\uplus$-gate. Let $W=W(e_{1})+W(e_{2})$. Since $u$ is disjoint, given $\tau\in\mathit{rel}(C_{u})$, either $\tau\in\mathit{rel}(C_{u_{1}})$ or $\tau\in\mathit{rel}(C_{u_{2}})$ but not both. It follows:

[TABLE]

by definition of $\omega_{e}$. Observe that by induction we have $W(e_{1})=\sum_{\tau\in\mathit{rel}(C_{u_{1}})}\omega_{e_{1}}(\tau_{1})$ and $W(e_{2})=\sum_{\tau\in\mathit{rel}(C_{u_{2}})}\omega_{e_{2}}(\tau_{2})$. Thus, by taking the constants out and using this identity, it follows:

[TABLE]

Case 2: Assume that $u$ is a $\times$-gate. Applying the definition of $\omega_{e}$, we get:

[TABLE]

Remember that by definition of $\{\uplus,\times\}$-circuits, $\mathit{rel}(C_{u})=\mathit{rel}(C_{u_{1}})\times\mathit{rel}(C_{u_{2}})$. That is, $\tau\in\mathit{rel}(C_{u})$ if and only if $\tau_{1}:=\tau_{\mid{\mathsf{Attr}(C_{u_{1}})}}\in\mathit{rel}(C_{u_{1}})$ and $\tau_{2}:=\tau_{\mid{\mathsf{Attr}(C_{u_{2}})}}\in\mathit{rel}(C_{u_{2}})$. Thus, we can rewrite the last sum as:

[TABLE]

By taking the constant $W(e)$ out of the sum and observing that the sum is now separated into two independent terms, we have:

[TABLE]

By induction, $W(e_{i})=\sum_{\tau\in\mathit{rel}(C_{u_{i}})}\omega_{e_{i}}(\tau_{i})$ for $i=1,2$. Hence both sums of the last term are equal to $1$, which means that

[TABLE]

∎

We choose $\omega=\omega_{o_{r}}$ where $o_{r}$ is the output edge. Lemma 4.7 is however not enough to prove Theorem 4.6 as it only gives the equality $W(e)=\sum_{\tau\in\mathit{rel}(C,e)}\omega(\tau)$ for $e=o_{r}$. Fortunately we can show that it holds for every edge $e$ of the circuit. We actually prove a stronger property, that $\omega_{e}$ is, in some sense, a projection of $\omega$.

Given an edge $e=\langle u,v\rangle$ of $C$ and $\tau^{\prime}\in\mathit{rel}(C_{u})$, we denote by $\mathit{rel}(C,e,\tau^{\prime})$ the set of tuples $\tau$ of $\mathit{rel}(C,e)$ such that $\tau_{\mid{\mathsf{Attr}(C_{u})}}=\tau^{\prime}$. We prove the following:

Lemma 4.8**.**

For every $e=\langle u,v\rangle\in\mathit{Edges(C)}$, for every $\tau^{\prime}\in\mathit{rel}(C_{u})$,

[TABLE]

Proof.

The proof is by top-down induction on $C$.

Base case: We prove the result for $e=o_{r}=\langle u,v\rangle$. Let $\tau^{\prime}\in\mathit{rel}(C_{u})$. Because $u$ is the output gate, we have $\mathsf{Attr}(C_{u})=\mathsf{Attr}(C)$ and hence $\mathit{rel}(C,e,\tau^{\prime})=\{\tau^{\prime}\}$. Recall that $\omega=\omega_{o_{r}}$ by definition. In other words, $\omega_{e}(\tau^{\prime})=\omega(\tau^{\prime})=\sum_{\tau\in\mathit{rel}(C,e,\tau^{\prime})}\omega(\tau)$.

Inductive case: Now let $e=\langle u,v\rangle$ be an internal edge of $C$. Let $o_{1},\dots,o_{n}$ be the outgoing edges of $v$, $u^{\prime}$ be the only sibling of $u$ and let $e^{\prime}=\langle u^{\prime},v\rangle$. See Figure 7 for a schema of these notations. We fix $\tau^{\prime}\in\mathit{rel}(C_{u})$ and prove the desired equality.

Case 1: $v$ is a $\uplus$-gate. In this case, $\tau^{\prime}\in\mathit{rel}(C_{v})$. We claim that

[TABLE]

For left-to-right inclusion, let $\tau\in\mathit{rel}(C,e,\tau^{\prime})$. By definition, its proof tree $\mathsf{T}_{C}(\tau)$ contains $e$. Since $\mathsf{T}_{C}(\tau)$ is connected by Proposition 4.1, $\mathsf{T}_{C}(\tau)$ has to contain one edge of $\mathsf{Out}(v)$. The disjointness of the right-side union is a direct consequence of Proposition 4.2. For the right-to-left inclusion, fix $o\in\mathsf{Out}(v)$ and let $\tau\in\mathit{rel}(C,o,\tau^{\prime})$. By definition, its proof tree $\mathsf{T}_{C}(\tau)$ contains $o$, thus it also contains the vertex $v$. Now recall that $\tau_{\mid{\mathsf{Attr}(C_{u})}}=\tau^{\prime}\in\mathit{rel}(C_{u})$. Thus, by definition of proof trees, $u$ is also in $\mathsf{T}_{C}(\tau)$. In other words, $\tau\in\mathit{rel}(C,e,\tau^{\prime})$. Using this equality, we have:

[TABLE]

Since, by induction, we have that for every $o\in\mathsf{Out}(v)$, $\omega_{o}(\tau^{\prime})=\sum_{\tau\in\mathit{rel}(C,o,\tau^{\prime})}\omega(\tau)$.

Assume first that $\sum_{o\in\mathsf{Out}(v)}W(o)=0$. In this case, by definition, for every $o$, $\omega_{o}(\tau^{\prime})=0$. Since $W$ is sound however, we also have $W(e)=0$, which implies by Lemma 4.7, that $\omega_{e}(\tau^{\prime})=0$ as well. In this case, $\sum_{\tau\in\mathit{rel}(C,e,\tau^{\prime})}\omega(\tau)=0=\omega_{e}(\tau^{\prime})$ which is the induction hypothesis.

Now assume that $\sum_{o\in\mathsf{Out}(v)}W(o)\neq 0$. We can thus apply the definition of $\omega_{o}(\tau^{\prime})={W(o)\over W(e)+W(e^{\prime})}\omega_{e}(\tau^{\prime})$ in the last sum. It gives

[TABLE]

where the last equality follows from the fact that $W$ is sound and thus the ratio is $1$.

Case 2: $v$ is a $\times$-gate. Similarly as before, we have:

[TABLE]

For left-to-right inclusion, let $\tau\in\mathit{rel}(C,e,\tau^{\prime})$. By definition, its proof tree $\mathsf{T}_{C}(\tau)$ contains $e$. Since $\mathsf{T}_{C}(\tau)$ is connected by Proposition 4.1, $\mathsf{T}_{C}(\tau)$ has to contain one edge $o$ of $\mathsf{Out}(v)$. Thus, $\tau\in\mathit{rel}(C,o,\tau^{\prime}\times\tau_{\mid{\mathsf{Attr}(C_{u^{\prime}})}})$.

The disjointness of the right-side union is a direct consequence of Proposition 4.2. For the right-to-left inclusion, fix $o\in\mathsf{Out}(v)$ and $\tau^{\prime\prime}\in\mathit{rel}(C_{u^{\prime}})$. Let $\tau\in\mathit{rel}(C,o,\tau^{\prime}\times\tau^{\prime\prime})$. By definition, its proof tree $\mathsf{T}_{C}(\tau)$ contains $o$, thus it also contains the vertex $v$ and by definition of proof trees, $u$ is also in $\mathsf{T}_{C}(\tau)$. And since $\tau_{\mid{\mathsf{Attr}(C_{u})}}=\tau^{\prime}$, $\tau\in\mathit{rel}(C,e,\tau^{\prime})$. Using this equality, we have:

[TABLE]

Since, by induction, we have that for every $o\in\mathsf{Out}(v)$, $\omega_{o}(\tau^{\prime}\times\tau^{\prime\prime})=\sum_{\tau\in\mathit{rel}(C,o,\tau^{\prime}\times\tau^{\prime\prime})}\omega(\tau)$.

Assume first that $\sum_{o\in\mathsf{Out}(v)}W(o)=0$. In this case, by definition, for every $o$ and $\tau^{\prime\prime}$, $\omega_{o}(\tau^{\prime}\times\tau^{\prime\prime})=0$. Since $W$ is sound however, we also have $W(e)=0$, which implies by Lemma 4.7, that $\omega_{e}(\tau^{\prime})=0$ as well. In this case, $\sum_{\tau\in\mathit{rel}(C,e,\tau^{\prime})}\omega(\tau)=0=\omega_{e}(\tau^{\prime})$ wich is the induction hypothesis.

Now assume that $\sum_{o\in\mathsf{Out}(v)}W(o)\neq 0$. We can thus apply the definition of $\omega_{o}(\tau^{\prime}\times\tau^{\prime\prime})=W(o){\omega_{e}(\tau^{\prime})\over W(e)}{\omega_{e^{\prime}}(\tau^{\prime\prime})\over W(e^{\prime})}$ in the last sum. It gives

[TABLE]

Since $W$ is sound, $\sum_{o\in\mathsf{Out}(v)}W(o)=W(e)$. Moreover, by Lemma 4.7, $\sum_{\tau^{\prime\prime}\in\mathit{rel}(C_{u^{\prime}})}\omega_{e^{\prime}}(\tau^{\prime\prime})=W(e^{\prime})$. Thus, the last sum equals to $\omega_{e}(\tau^{\prime})$ which concludes the proof.

∎

Theorem 4.6 b is now an easy consequence of Lemma 4.7 and Lemma 4.8:

[TABLE]

Indeed, $\mathit{rel}(C,e)=\biguplus_{\tau^{\prime}\in\mathit{rel}(C_{u})}\mathit{rel}(C,e,\tau^{\prime})$. For the left-to-right inclusion, if $\tau\in\mathit{rel}(C,e)$ then by definition, $\tau^{\prime}=\tau_{\mid{\mathsf{Attr}(C_{u})}}\in\mathit{rel}(C_{u})$ and thus $\tau\in\mathit{rel}(C,e,\tau^{\prime})$. The other inclusion follows by definition since $\mathit{rel}(C,e,\tau^{\prime})\subseteq\mathit{rel}(C,e)$ for every $\tau^{\prime}\in\mathit{rel}(C_{u})$.

5 Consequences and extension of the result

5.1 Effective reconstruction of solutions

As it is illustrated in the motivating example of the introduction, one may not only be interested in the optimal value of a CAS-LP but also on finding an optimal solution. As the size of a factorized relation may be too big, it may not be tractable to output the entire optimal solution. However, the construction used in the proof of Theorem 4.6 to go from an edge-weighting to a tuple-weighting is effective in the sense that given an edge-weighting $W$ of a $\{\uplus,\times\}$-circuit $C$ and $\tau\in\mathit{rel}(C)$, one can compute $\omega(\tau)$ in time polynomial in $|C|$ (where $\omega$ is the tuple-weighting inducing $W$ given by Theorem 4.6). It is indeed enough to compute the partial tuple weightings $\omega_{e}$ presented in Section 4.4 in a bottom-up induction on $C$.

5.2 Tractability for conjunctive queries

By connecting Theorem 3.1 and Theorem 2.2, we directly get the tractability of solving bounded fractional hypertree width quantifier free conjunctive queries:

Corollary 5.1**.**

Given a CAS-LP $L$, a quantifier-free conjunctive query $Q$, a hypertree decomposition of $Q$ of fractional hypertreewidth $k$ and a database $\mathbb{D}$, one can compute the optimal value of $L(Q(\mathbb{D}))$ in polynomial time.

The exact complexity of Corollary 5.1 depends on the runtime of the linear program solver that is used that also highly depends on the structure of the linear program. We do not know whether we could directly solve the linear program on the circuit without calling the solver and have better complexity bounds.

The complexity also depends on the fact that we are given a good hypertree decomposition in the input. Computing the best decomposition for fractional hypertree width is known to be an $\mathsf{NP}$-hard problem [FGP18] but it can be approximated in polynomial time to a cubic factor [Mar10] which is enough if one is interested only in theoretical tractability. In practice however, our algorithm would perform the best when the circuit is small, be it computed from hypertree decompositions or other techniques.

Our result does not directly apply to quantified conjunctive queries. Indeed, existentially or universally projecting variables in a $\{\uplus,\times\}$-circuit may lead to an exponential blow-up of its size or the disjointness of unions may be lost. However, for existential quantification, we do not need to actually project the variables in the circuit to compute the optimal value. Given a relation $R$ on attribute $\mathsf{X}$ and $Z\subseteq\mathsf{X}$, we denote by $\exists Z.R=\{\tau|_{\mathsf{X}\setminus Z}\mid\tau\in R\}$. We have the following:

Theorem 5.2**.**

Let $L$ be a CAS-LP on attributes $\mathsf{X}\setminus Z$ and domain $\mathsf{D}$. Let $R$ be a relation on attributes $\mathsf{X}$ and domain $\mathsf{D}$. $L(R)$ and $L(\exists Z.R)$ have the same optimal value.

Proof.

We show how one can transform a solution of $L(R)$ into a solution of $L(\exists Z.R)$ with the same value and vice-versa.

We introduce the notation for $\tau^{\prime}\in\exists Z.R$:

[TABLE]

Let $\omega:R\rightarrow\mathbb{R}_{+}$ be a solution of $L(R)$. Given $\tau^{\prime}\in\exists Z.R$, we define $\omega^{\prime}(\tau^{\prime}):=\sum_{\tau^{\prime\prime}\in\mathit{Ext}(\tau^{\prime})}\omega(\tau^{\prime}\cdot\tau^{\prime\prime})$. It is easy to see that $\omega^{\prime}$ is a solution of $L(\exists Z.R)$ since $\omega^{\prime}(S_{x,d})=\omega(S_{x,d})$ for every $x\in\mathsf{X}\setminus Z$ and $d\in\mathsf{D}$.

Similarly, if $\omega^{\prime}:\exists Z.R\rightarrow\mathbb{R}_{+}$ is a solution of $L(\exists Z.R)$, we define for $\tau\in R$, $\omega(\tau):={\omega^{\prime}(\tau^{\prime})\over\#\mathit{Ext}(\tau^{\prime})}$ where $\tau^{\prime}=\tau|_{\mathsf{X}\setminus Z}$. Again, it is easy to see that $\omega(S_{x,d})=\omega^{\prime}(S_{x,d})$ which concludes the proof. ∎

Theorem 5.2 implies that one can still solve CAS-LPs on relations given as existential projection of a $\{\uplus,\times\}$-circuit. In other words, if a conjunctive query $Q$ is of the form $\exists Z.Q^{\prime}$ and $Q^{\prime}$ has bounded fractional hypertree width, then we can still solve CAS-LPs on $Q(\mathbb{D})$ for a given database $\mathbb{D}$ in polynomial time by transforming $Q^{\prime}$ into a $\{\uplus,\times\}$-circuit and solving the CAS-LP on it directly.

6 Conclusion

In this paper, we have initiated the study of a new kind of natural aggregation tasks: solving optimization problems whose variables are the answers of a database query. We isolated an interesting class of linear programs that is intractable for conjunctive queries in general but for which we provide an algorithm that runs in polynomial time when the fractional hypertreewidth of the conjunctive query is bounded. Our technique relies on factorized representation of the answer set of the conjunctive query.

There are many possible future directions based on the techniques we present in this paper. A first step would be to generalize our methods to larger classes of linear programs, allowing more complex meta-variables than the CAS variables that we use in this paper. Another limitation of our work that we would like to improve is that it can currently only handle linear programs having positive values for technical reasons in the way our proof is constructed. Building on this idea, a very exciting yet ambitious program is to generalize and integrate our methods in a larger framework resembling GNU Math Prog or AMPL augmented with database queries on which we could act to generate smaller linear programs than the naive grounding.

Finally, we would like to investigate applications of our result and methods to existing techniques in data mining and databases. A first interesting step would be to use our algorithm to compute the $s$-measure of [WRF13] in practice for some real life queries. We would like to try to run the algorithm on benchmarks from data mining and see how they compare with the bruteforce approach of generating the complete ground linear program. We already tried a basic implementation of our algorithm on syntactical data to see if the linear program solver scales but did not yet consider any real benchmark.

Moreover, it would be interesting to investigate more deeply connections with other works using Binary or Mixed-Integer Programming to solve database problems. In [KPT13] the authors use binary programming to answer queries on inconsistent databases. They do so by generating binary programs with two types of constraints. The first type of constraints encodes the possible repairs of the database by limiting the total weight of the sum of the tuples sharing a value for a primary key. These are constraints over CAS variables in our framework and can thus be handled seamlessly. The second type of constraints deals with witnesses of the potential answers. We might be able to handle theses constraints by adding some information about the witnesses to the circuit.

In [MS12] the authors use Mixed-Integer Programming to answer how-to queries expressed in the ”Tiresias Query Language” (TiQL) which is based on datalog. The constraints of their Mixed-Integer Programs appear to be quite different at first glance from what our framework is able to handle. However the constraints are generated using the provenance of the tuples, a notion which is closely related to the circuits we exploit to solve CAS-LPs more efficiently so it might be possible to exploit the structure of the circuits in a different way to answer TiQL queries more efficiently.

Acknowledgment

This work was partially supported by the French Agence Nationale de la Recherche, AGGREG project reference ANR-14-CE25-0017-01, Headwork project reference ANR-16-CE23-0015 and by a grant of the Conseil Régional Hauts-de-France. The project DATA, Ministère de l’Enseignement Supérieur et de la Recherche, Région Nord-Pas de Calais and European Regional Development Fund (FEDER) are acknowledged for supporting and funding this work. We also thank Rui Castro, Sylvain Salvati, Sophie Tison and Yuyi Wang for fruitful discussions and anonymous reviewers of a previous version of this paper for their helpful comments.