General Bounds on Satisfiability Thresholds for Random CSPs via Fourier Analysis

TL;DR

This paper introduces general bounds for the satisfiability thresholds of random boolean CSPs using Fourier analysis, applicable to any fixed constraint function, and demonstrates their effectiveness through theoretical and experimental validation.

Contribution

It provides the first fully general bounds on CSP thresholds based on Fourier spectra, applicable to any constraint function, advancing understanding of phase transitions in random CSPs.

Findings

Bounds are within a constant factor of the true thresholds for many CSPs.

Bounds can be computed from the Fourier spectrum of constraints.

Experimental results validate the bounds' effectiveness.

Abstract

Random constraint satisfaction problems (CSPs) have been widely studied both in AI and complexity theory. Empirically and theoretically, many random CSPs have been shown to exhibit a phase transition. As the ratio of constraints to variables passes certain thresholds, they transition from being almost certainly satisfiable to unsatisfiable. The exact location of this threshold has been thoroughly investigated, but only for certain common classes of constraints. In this paper, we present new bounds for the location of these thresholds in boolean CSPs. Our main contribution is that our bounds are fully general, and apply to any fixed constraint function that could be used to generate an ensemble of random CSPs. These bounds rely on a novel Fourier analysis and can be easily computed from the Fourier spectrum of a constraint function. Our bounds are within a constant factor of the exact threshold location for many well-studied random CSPs. We demonstrate that our bounds can be easily instantiated to obtain thresholds for many constraint functions that had not been previously studied, and evaluate them experimentally.

Figures

Equations

$${\sigma }_{I,s}=(s_1{\sigma }_{i_1},\dots ,s_k{\sigma }_{i_k})$$

$$\underset{n\to \infty }{\lim }\mathrm{Pr}[C_f(n,rn)\text{is satisfiable}]=\{\begin{array}{ll}1\text{if}& r<r_{f,\text{sat}}\\ 0\text{if}& r>r_{f,\text{unsat}}\end{array}$$

$$\mathrm{Pr}[C_f(n,rn)\text{is satisfiable}]>C\text{for all}r<r_{f,\text{low}}$$

$$\underset{n\to \infty }{\lim }\mathrm{Pr}[C_f(n,rn)\text{is satisfiable}]=1\text{for all}r<r_{f,\text{low}}$$

$$\underset{n\to \infty }{\lim }\mathrm{Pr}[C_f(n,rn)\text{is satisfiable}]=0\text{for all}r>r_{f,\text{up}}$$

$$\hat{f}(S)=⟨f,{\chi }_S⟩=\frac{1}{2^k}\sum _{u\in \{-1,1{\}}^k}f(u){\chi }_S(u)$$

$$r_{f,\text{up}}=\frac{\log 2}{\log 1/\hat{f}(\varnothing )}<\frac{\log 2}{1-\hat{f}(\varnothing )}$$

$$\displaystyle A=\left[\begin{array}[]{r r r r r r r}1&1&1&1&-1&-1&-1\\ 1&1&-1&-1&1&1&-1\\ 1&-1&1&-1&1&-1&1\end{array}\right]$$

$$r_{f,\text{low}}=\frac{1}{2}\frac{c}{1-c}\text{ where }c=\hat{f}(\varnothing )-\frac{1^T{A}^{+}A1}{2^k}$$

$$f(u_1,\dots ,u_{3a})=$$

$$f_{a-\text{MAJ}}(f_{3-\text{MAJ}}(u_1,u_2,u_3),\dots ,$$

$$f_{3-\text{MAJ}}(u_{3a-2},u_{3a-1},u_{3a}))$$

$$f(u_1,\dots ,u_{ab})=$$

$$f_{{\text{OR}}_b}(f_{{\text{XOR}}_a}(u_1,\dots ,u_a),\dots ,$$

$$f_{{\text{XOR}}_a}(u_{ab-a+1},\dots ,u_{ab}))$$

$$\displaystyle B=\left[\begin{array}[]{r r r}7/8&-1/8&-1/8\\ -1/8&7/8&-1/8\\ -1/8&-1/8&7/8\end{array}\right]$$

$$Pr[X\ne 0]=\mathrm{Pr}[|X|\ne 0]\ge \frac{\text{E}[|X|{]}^2}{\text{E}[X^2]}\ge \frac{\text{E}[X{]}^2}{\text{E}[X^2]}$$

$$X=\sum _{\sigma \in \{-1,1{\}}^n}\prod _{c\in C_f(n,rn)}w(\sigma ,c)$$

$$\text{E}[X{]}^2=2^{2n}(\hat{w}(\varnothing {)}^2{)}^{rn}$$

$$\text{E}[X]$$

$$=\sum _{\sigma \in \{-1,1{\}}^n}\text{E}[w({\sigma }_{I,s}){]}^{rn}$$

$$\mathrm{Pr}[{\sigma }_{I,s}=u]=\frac{1}{2^k}$$

$$\text{E}[w({\sigma }_{I,s})]$$

$$=\frac{1}{2^k}\sum _{u\in \{-1,1{\}}^k}w(u)$$

$$=\hat{w}(\varnothing )$$

$$\text{E}[X^2]=2^n\sum _{j=0}^n\left(\genfrac{}{}{0ex}{}{n}{j}\right)g_w(j/n{)}^{rn}$$

$$\text{E}[X{]}^2=2^{2n}{g}_w(1/2{)}^{rn}$$

$$S_n=\sum _{j=0}^n\left(\genfrac{}{}{0ex}{}{n}{j}\right)\varphi (j/n{)}^n$$

$$B\psi ({\alpha }_{\max }{)}^n\le S_n\le C\psi ({\alpha }_{\max }{)}^n$$

$${\psi }_r(1/2{)}^n=2^n(g_w(1/2{)}^r{)}^n=\text{E}[X{]}^2/2^n$$

$$\sum _{j=0}^n\left(\genfrac{}{}{0ex}{}{n}{j}\right){\varphi }_r(j/n{)}^n=\text{E}[X^2]/2^n$$

$$g_w^{\prime }(1/2)=2\sum _{S\subseteq [k]:|S|=1}\hat{w}(S{)}^2=0$$

$$r\le \frac{1}{2}\frac{\hat{w}(\varnothing {)}^2}{{\sum }_{S:|S|\ge 2}\hat{w}(S{)}^2}$$

$$w(u)=\lambda (u)f(u)$$

Full text

In this paper, we present new bounds for the location of these thresholds in boolean CSPs. Our main contribution is that our bounds are fully general, and apply to any fixed constraint function that could be used to generate an ensemble of random CSPs. These bounds rely on a novel Fourier analysis and can be easily computed from the Fourier spectrum of a constraint function. Our bounds are within a constant factor of the exact threshold location for many well-studied random CSPs. We demonstrate that our bounds can be easily instantiated to obtain thresholds for many constraint functions that had not been previously studied, and evaluate them experimentally.

1 Introduction

Constraint satisfaction problems (CSPs) are widely used in AI, with applications in optimization, control, and planning (?). While many classes of CSPs are intractable in the worst case (?), many real-world CSP instances are easy to solve in practice (?). As a result, there has been significant interest in understanding the average-case complexity of CSPs from multiple communities, such as AI, theoretical computer science, physics, and combinatorics (?).

Random CSPs are an important model for studying the average-case complexity of CSPs. Past works have proposed several distributional models for random CSPs (?; ?). An interesting feature arising from many models is a phase transition phenomenon that occurs as one changes the ratio of number of constraints, $m$, to the number of variables, $n$. Empirical results (?) show that for many classes of CSPs, randomly generated instances are satisfiable with probability near 1 when $m/n$ is below a certain threshold. For $m/n$ larger than a threshold, the probability of satisfiability is close to 0. The statistical physics, computer science, and mathematics communities have focused much attention on identifying these threshold locations (?; ?).

Phase transitions for common classes of CSPs such as $k$-SAT and $k$-XORSAT are very well-studied. For $k$-SAT, researchers struggled to find tight lower bounds on the satisfiability threshold until the breakthrough work of ?, which provided lower bounds a constant factor away from the upper bounds. Later works closed this gap for $k$-SAT (?; ?). More recently, ? (?) also studied the satisfiability threshold for a more general CSP class, namely $k$-CNF-XOR, where both $k$-SAT and $k$-XORSAT constraints can be used. The results and analyses from these works, however, are all specific to the constraint classes studied.

In this paper, we provide new lower bounds on the location of the satisfiability threshold that hold for general boolean CSP classes. We focus on the setting where CSPs are generated by a single constraint type, though our analysis can extend to the setting with uniform mixtures of different constraint functions. We extend techniques from (?) and build on (?), which proposes a distributional model for generating CSPs and provides lower bounds on the satisfiability threshold for these models. The significance of our work is that our bounds hold for all functions that could be used to generate random CSP instances. The lower bounds from (?) are also broadly applicable, but they are looser than ours because they do not depend on constraint-specific properties. Our lower bounds are often tight (within a constant factor of upper bounds for many CSP classes) because they depend on specific properties of the Fourier spectrum of the function used to generate the random CSPs. Since these properties are simple to compute for any constraint function, our lower bounds are broadly applicable too.

The Fourier analysis of boolean functions (?) will be vital for obtaining our main results. Expressing functions in the Fourier basis allows for clean analyses of random constraints (?; ?; ?). Our use of Fourier analysis is inspired by the work of ? (?), who analyze the Fourier spectra of random hash functions used as constraints in CSP-based model counting. We show that the Fourier spectrum of our constraint-generating function controls the level of spatial correlation in the set of satisfying assignments to the random CSP. If the Fourier spectrum is concentrated on first and second order coefficients (corresponding to “low frequencies”), this correlation will be very high, roughly increasing the variance of the number of solutions to a random CSP and decreasing the probability of satisfiability. In related work, ? (?) also use Fourier analysis to provide tight thresholds in the case where odd Fourier coefficients are all zero.

2 Notation and Preliminaries

In this section, we will introduce the preliminaries necessary for presenting our main theorem. First, we formally define our distribution for generating random CSPs, inspired from (?).

We will use $n$ and $m$ to denote the number of variables and number of constraints in our CSPs, respectively. We will also let $f:\{-1,1\}^{k}\rightarrow\{0,1\}$ denote a binary function and refer to $f$ as our constraint function. Often we use the term “solution set of $f$” to refer to the set $\{u:f(u)=1\}$. Using the constraint function $f$, we create constraints by applying $f$ to a signed subset of $k$ variables.

Definition 1** (Constraint).**

Let $I=(i_{1},\ldots,i_{k})$ be an ordered tuple of $k$ indices in $[n]$, and let $s$ be a sign vector from $\{-1,1\}^{k}$. Given a vector $\sigma\in\{-1,1\}^{n}$, we will define the vector $\sigma_{I,s}$ of size $k$ as follows:

[TABLE]

Now we can denote the application of $f$ to these indices by $f_{I,s}(\sigma)=f(\sigma_{I,s})$. We call $f_{I,s}$ a constraint, and we say that $\sigma\in\{-1,1\}^{n}$ satisfies the constraint $f_{I,s}$ if $f_{I,s}(\sigma)=1$.

The definition of a CSP generated from $f$ follows.

Definition 2** (CSP generated from $f$).**

We will represent a CSP with $m$ constraints and $n$ variables generated from $f$ as a collection of constraints $C_{f}(n,m)=\{f_{I_{1},s_{1}},\ldots,f_{I_{m},s_{m}}\}$. Then $\sigma\in\{-1,1\}^{n}$ satisfies $C_{f}(n,m)$ if $\sigma$ satisfies $f_{I_{j},s_{j}}$ for $j=1,\ldots,m$.

Example 1** (3-SAT).**

Let $f:\{-1,1\}^{3}\rightarrow\{1,0\}$ where $f(u)=0$ for $u=(-1,-1,-1)$ and $f(u)=1$ for all other $u$. Then $f$ is the constraint function for the 3-SAT problem.

With these basic definitions in place, we are ready to introduce the model for random CSPs.

2.1 Random CSPs

We discuss our model for randomly generating $C_{f}(n,m)$, and formally define a “satisfiability threshold.”

To generate instances of $C_{f}(n,m)$, we simply choose $I_{1},\ldots,I_{m}$ and $s_{1},\ldots,s_{m}$ uniformly at random. For completeness, we sample without replacement, i.e. there are no repeated variables in a constraint, and no duplicate constraints. However, sampling with replacement does not affect final results. For the rest of this paper we abuse notation and let $C_{f}(n,m)$ denote a randomly generated CSP instance following this model.

Now we formally discuss satisfiability thresholds. We let $r=m/n$. For many constraint functions $f$, there exist thresholds $r_{f,\text{sat}}$ and $r_{f,\text{unsat}}$ such that

[TABLE]

In general, it is unknown whether $r_{f,\text{sat}}=r_{f,\text{unsat}}$, but for some problems such as $k$-SAT (for large $k$) and $k$-XORSAT, affirmative results exist (?; ?). If $r_{f,\text{sat}}=r_{f,\text{unsat}}$, then we say that the random CSP $C_{f}(n,m)$ exhibits a sharp threshold in $m/n$.

We are concerned with finding lower bounds $r_{f,\text{low}}$ on $r_{f,\text{unsat}}$ such that there exists a constant $C>0$ independent of $n$ so that for sufficiently large $n$,

[TABLE]

For CSP classes with a sharp threshold, (2) implies that

[TABLE]

We also wish to find upper bounds $r_{f,\text{up}}$ such that

[TABLE]

For an example of these quantities instantiated on a concrete example, refer to the experiments in Section 5.

We provide a value for $r_{f,\text{up}}$ which was derived earlier in (?). ? (?) and ? (?) provide methods for obtaining tighter upper bounds, but the looser values that we use are sufficient for showing that $r_{f,\text{low}}$ is on the same asymptotic order as $r_{f,\text{unsat}}$ for many choices of $f$.

The bound $r_{f,\text{low}}$ depends on both the symmetry and size of the solution set of $f$. The more assignments $u\in\{-1,1\}^{k}$ such that $f(u)=1$, the more likely it is that each constraint is satisfied. Increased symmetry reduces the variance in the number of solutions to $C_{f}(n,rn)$, so solutions are more spread out among possible CSPs in our class and the probability that $C_{f}(n,rn)$ will have a solution is higher. We will formally quantify this symmetry in terms of the Fourier spectrum of $f$, which we introduce next.

2.2 Fourier Expansion of Boolean Functions

We discuss basics of Fourier analysis of boolean functions. For a detailed review, refer to (?). We define the vector space $\mathcal{F}_{k}$ of all functions mapping $\{-1,1\}^{k}$ to $\mathbb{R}$. The set $\mathcal{F}_{k}$ has the inner product $\langle f_{1},f_{2}\rangle=\sum_{u\in\{-1,1\}^{k}}f_{1}(u)f_{2}(u)/2^{k}$ for any $f_{1},f_{2}\in\mathcal{F}_{k}$.

This inner product space has orthonormal basis vectors $\chi_{S}$, where the parity functions $\chi_{S}$ follow $\chi_{S}(u)=\prod_{i\in S}u_{i}$ for all $S\subseteq[k]$, subsets of the $k$ indices. Because $(\chi_{S})_{S\subseteq[k]}$ forms an orthonormal basis, if we write

[TABLE]

then we can write $f$ as a linear combination of these vectors: $f=\sum_{S\subseteq[k]}\hat{f}(S)\chi_{S}$. We note that when $S=\emptyset$, the empty set, $\hat{f}(\emptyset)$ is simply the average of $f$ over $\{-1,1\}^{k}$. We will refer to the coefficients $(\hat{f}(S))_{S\subseteq[k]}$ as the Fourier spectrum of $f$. Since these coefficients are well-studied in theoretical computer science (?), the Fourier coefficients of many boolean functions are easily obtained.

Example 2** (3-SAT).**

For $3$-SAT, $\hat{f}(\emptyset)=7/8$. $\hat{f}(\{1\})=\hat{f}(\{2\})=\hat{f}(\{3\})=1/8$, and $\hat{f}(\{1,2\})=\hat{f}(\{2,3\})=\hat{f}(\{1,3\})=-1/8$.

Representing $f$ in the Fourier bases will facilitate our proofs, providing a simple way to express expectations over our random CSPs. The Fourier spectrum can also provide a measure of “symmetry” in $f$ - if some values of $\hat{f}(S)$ are high where $|S|=1$, then satisfying assignments to $f$ are more skewed in the variable corresponding to $S$. We will show how this impacts satisfiability in Section 3.2.

3 Main Results

We provide simple formula for $r_{f,\text{up}}$. A similar result is in (?), and the full proof is in the appendix.

Proposition 1**.**

For all constraint functions $f$, let

[TABLE]

If $r\geq r_{f,\text{up}}$, $\lim_{n\rightarrow\infty}\Pr[C_{f}(n,rn)\ \text{is satisfiable}]=0$.

Proof Sketch.

We compute the expected solution count for $C_{f}(n,rn)$. The expected solution count will scale with $\hat{f}(\emptyset)$, since $2^{k}\hat{f}(\emptyset)$ is simply the number of $u\in\{-1,1\}^{k}$ where $f(u)=1$ and therefore governs how easily each constraint will be satisfied. If $r>r_{f,\text{up}}$, the expected solution count converges to 0 as $n\rightarrow\infty$, so Markov’s inequality implies that the probability that a solution exists goes to 0. ∎

Next, we will present our value for $r_{f,\text{low}}$. First, some notation: let $U=\{u\in\{-1,1\}^{k}:f(u)=1\}$, and let $A$ be the $k\times|U|$ matrix whose columns are the elements of $u$. We will use $A^{+}$ to denote the Moore-Penrose pseudoinverse of $A$. For a reference on this, see (?). Finally, let $\boldsymbol{1}$ be the $|U|$-dimensional vector of 1’s.

Example 3** ($3$-SAT).**

For $3$-SAT, $k=3$ and $|U|=7$, and we can write $A$ as follows (up to permutation of its columns):

[TABLE]

where columns of $A$ satisfy the $3$-SAT constraint function.

The following main theorem provides the first computable equation for obtaining lower bounds that are specific to the constraint-generating function $f$.

Theorem 1**.**

For all constraint functions $f$, let

[TABLE]

If $r<r_{f,\text{low}}$, then there exists a constant $C>0$ such that $\lim_{n\rightarrow\infty}\Pr[C_{f}(n,rn)\ \text{is satisfiable}]>C$.

The lower bound $r_{f,\text{low}}$ is an increasing function of $c$ which is dependent on two quantities. First, with higher values of $\hat{f}(\emptyset)$, $C_{f}(n,rn)$ will have more satisfying assignments on average, so $c$ and the threshold value will be higher. Second, $c$ depends on the level of symmetry in the solution set of $f$, which we will show is connected to the Fourier spectrum of $f$. We explain this dependence in Section 3.2.

In comparison, ? (?) obtain lower bounds which depend only on the arity of $f$. While we cannot make an exact comparison because ? use a different random CSP ensemble, for reference, they provide the general lower bound of $1/(ke^{k}-k)$ expected constraints per variable for functions of arity $k$. Our bounds are much tighter because of their specificity while remaining simple to compute. To demonstrate, we instantiate our bounds for some example constraint functions in Figure 1. Whereas their bounds are exponentially decreasing in $k$, our bounds are constant or increasing in $k$ for the functions shown.

3.1 Constraint Functions in Figure 1

We define the constraint functions in Figure 1. Unless specified otherwise, they will be in the form $f:\{-1,1\}^{k}\rightarrow\{0,1\}$.

$k$-SAT: $f(u)=0$ if $u$ is the all negative ones vector, and $f(u)=1$ otherwise. 2. 2.

$k$-XORSAT: $f(u)=\mathbbm{1}(\chi_{[k]}(u)=-1)$ 3. 3.

$k$-NAESAT: $f(u)=0$ if $u$ is the all negative ones or all ones vector, and $f(u)=1$ otherwise. 4. 4.

$k$-MAJORITY: Defined when $k$ is odd, $f(u)=1$ if more than half of the variables of $u$ are $1$. 5. 5.

$a$-MAJ $\otimes$ $3$-MAJ: Defined when $a$ is odd, where $f:\{-1,1\}^{3a}\rightarrow\{0,1\}$. Defined as the composition of $a$-MAJORITY on $a$ groups of $3$-MAJORITY, as follows:

[TABLE] 6. 6.

$k$-MOD-3: $f(u)=1$ when the number of 1’s in $u$ is divisible by $3$, and 0 otherwise. 7. 7.

OR${}_{b}\otimes$ XORa: In this case, $f:\{-1,1\}^{ab}\rightarrow\{0,1\}$, and $f$ is the composition of a OR over $b$ groups of XORs over $a$ variables, as follows:

[TABLE]

While the last four constraint functions have not been analyzed much in the existing CSP literature, these types of general constraints are of practical interest because of (?), which performs probabilistic inference by solving CSPs based on arbitrary hash functions. For example, ? (?) show that MAJORITY constraints are effective in practice for solving probabilistic inference problems.

3.2 Connecting Bounds with Fourier Spectrum

We explain how the Fourier spectrum can help us interpret Theorem 1. We first show the connection between $c$ and the Fourier spectrum. Let $\hat{f}_{S:|S|=1}$ be the $k$-dimensional vector whose entries are Fourier coefficients of $f$ for size 1 sets. Let $B$ be the $k\times k$ matrix with diagonal entries $B_{ii}=\hat{f}(\emptyset)$ and off-diagonal entries $B_{ij}=\hat{f}(\{i,j\})$ for $i\neq j$.

Example 4** (3-SAT).**

Following the coefficients in Example 2, for $3$-SAT, $\hat{f}_{S:|S|=1}=(1/8,1/8,1/8)$ and

[TABLE]

Lemma 1**.**

When the rows of $A$ are linearly independent, $c=\hat{f}(\emptyset)-\hat{f}_{S:|S|=1}^{T}B^{-1}\hat{f}_{S:|S|=1}$.

From this lemma, we see that larger values of $c$ correspond to smaller $\hat{f}_{S:|S|=1}$. These terms will measure the amount of “symmetry” in the solution set for $f$. The matrix $B$ and vectors $\hat{f}_{S:|S|=1}$ are easily obtained for many $f$ since Fourier coefficients are well-studied (?).

Figure 1 shows how $\hat{f}(\emptyset)-c$ and $f$ relate. Since $k$-SAT has a mostly symmetric solution set, $\hat{f}_{k-\text{SAT}}(\emptyset)-c=O(k/2^{2k})$ since $c=1-2^{-k}-O(k/2^{2k})$, which is small compared to $\hat{f}_{k-\text{SAT}}(\emptyset)$. The solution set of $k$-NAESAT is completely symmetric as if $f_{k-\text{NAESAT}}(x)=1$, then $f_{k-\text{NAESAT}}(-x)=1$. Thus, $f_{k-\text{NAESAT}}$ has 0 weight on Fourier coefficients for sets with odd size so we can compute that $\hat{f}_{k-\text{NAESAT}}(\emptyset)-c=0$. $k$-MAJORITY, however, is less symmetric, as shown by larger first order coefficients. Here, the bound in Figure 1 gives $\lim_{k\rightarrow\infty}\hat{f}_{k-\text{MAJORITY}}(\emptyset)-c=1/\pi$, which is large compared to $\hat{f}_{k-\text{MAJORITY}}(\emptyset)\approx 1/2$.

4 Proof Strategy

Our proof relies on the second moment method, which has been applied with great success to achieve lower bounds for problems such as $k$-SAT (?) and $k$-XORSAT (?). The second moment method is based on the following lemma, which can be derived using the Cauchy-Schwarz inequality:

Lemma 2**.**

Let $X$ be any real-valued random variable. Then

[TABLE]

If $X$ is only nonzero when $C_{f}(n,rn)$ has a solution, we obtain lower bounds on the probability that a solution exists by upper bounding $\text{E}[X^{2}]$. For example, we could let $X$ be the number of solutions to $C_{f}(n,rn)$. However, as shown in (?), this choice of $X$ fails in most cases. Whether two different assignments satisfy $C_{f}(n,rn)$ is correlated: if the assignments are close in Hamming distance and one assignment is satisfying, it is more likely that the other is satisfying as well. This will make $\text{E}[X^{2}]$ much larger than $\text{E}[X]^{2}$, so (4) will not provide useful information. Figure 2a demonstrates this failure for $k$-SAT. ? (?) show formally that the ratio $\text{E}[X]^{2}/\text{E}[X^{2}]$ will decrease exponentially (albeit at a slow rate). On the other hand, $k$-NAESAT is “symmetric”, so the second moment method works directly here. In the plot, $\text{E}[X]^{2}/\text{E}[X^{2}]$ for $3$-NAESAT stays above a constant. This also follows formally from our main theorem as well as (?). We formally define our requirements on symmetry in (13).

We circumvent this issue by weighting solutions to reduce correlations before applying the second moment method. As in (?), we use a weighting which factors over constraints in $C_{f}(n,rn)$ and apply the second moment method to the random variable

[TABLE]

where $C_{f}(n,rn)$ is a collection of constraints $\{f_{I_{1},s_{1}},\ldots,f_{I_{m},s_{m}}\}$ and the randomness in $X$ comes over the choices of $I_{j},s_{j}$. Now we can restrict our attention to constraint weightings of the form $w(\sigma,f_{I,s})=w(\sigma_{I,s})$. In the special case where $w(\sigma_{I,s})=f(\sigma_{I,s})$, $X$ will simply represent the number of solutions to $C_{f}(n,rn)$. In general, we require $w(\sigma_{I,s})=0$ whenever $f(\sigma_{I,s})=0$. This way, if $X\neq 0$, then $C_{f}(n,rn)$ must have a solution.

For convenience, we assume that the index sets $I_{1},\ldots,I_{m}$ are sampled with replacement. They are chosen uniformly from $[n]^{k}$. We also allow constraints to be identical. In the appendix, we justify why proofs in this setting carry over to the without-replacement setting in Section 2.1 and also provide full proofs to the lemmas presented below.

In this setting, we will compute the first and second moments of the $X$ chosen in (5) in terms of the Fourier spectrum of $w$.

Lemma 3**.**

The squared first moment of $X$ is given by

[TABLE]

Proof.

We can expand $\text{E}[X]$ as follows:

[TABLE]

where we used the fact that constraints are chosen independently. Now we claim that for any $u\in\{-1,1\}^{k}$,

[TABLE]

This follows from the fact that we choose $s$ uniformly over $\{-1,1\}^{k}$ and our definition of $\sigma_{I,s}$ in (1). Thus,

[TABLE]

Plugging back into (7) gives the desired result. ∎

Next, we will compute the second moment $\text{E}[X^{2}]$.

Lemma 4**.**

Let $g_{w}(\alpha)=\sum_{S\subseteq[k]}(2\alpha-1)^{|S|}\hat{w}(S)^{2}$. The second moment of $X$ is given by

[TABLE]

The function $g_{w}(\alpha)$ is similar to the noise sensitivity of a boolean function (?) and measures the correlation in the value of $w$ between two assignments $\sigma$, $\tau$ which overlap at $\alpha(\sigma,\tau)n$ locations. As a visual example, Figure 2b shows how $g_{w}(\alpha)$ changes for $k$-XORSAT with varying $k$. The key of our proof is showing that $\text{E}[w(\sigma_{I,s})w(\tau_{I,s})]=g_{w}(\alpha(\sigma,\tau))$ for a random constraint $f_{I,s}$.

We will now write $\text{E}[X]^{2}$ in terms of $g_{w}$. Since $g_{w}(1/2)=\hat{w}(\emptyset)^{2}$, plugging this into (6) gives us

[TABLE]

This motivates us to apply the following lemma from (?), which will allow us to translate bounds on $g_{w}(\alpha)$ into bounds on $\text{E}[X]^{2}/\text{E}[X^{2}]$:

Lemma 5**.**

Let $\phi$ be any real, positive, twice-differentiable function on $[0,1]$ and let

[TABLE]

Define $\psi$ on $[0,1]$ as $\psi(\alpha)=\frac{\phi(\alpha)}{\alpha^{\alpha}(1-\alpha)^{1-\alpha}}$. If there exists $\alpha_{\max}\in(0,1)$ such that $\psi(\alpha_{\max})>\psi(\alpha)$ for all $\alpha\neq\alpha_{\max}$, and $\psi^{\prime\prime}(\alpha_{\max})<0$, then there exist constants $B,C>0$ such that for all sufficiently large $n$,

[TABLE]

To apply the lemma, we can define $\phi_{r}(\alpha)=g_{w}(\alpha)^{r}$ and $\psi_{r}(\alpha)=\frac{\phi_{r}(\alpha)}{\alpha^{\alpha}(1-\alpha)^{1-\alpha}}$. Then from (9), we note that

[TABLE]

On the other hand, from (8),

[TABLE]

so if the conditions of Lemma 5 hold for $\alpha_{\max}=1/2$, we recover that $\text{E}[X]^{2}/\text{E}[X^{2}]\geq C$ for some constant $C>0$.

One requirement for $\psi_{r}(\alpha)$ to be maximized at $\alpha=1/2$ is that $\psi_{r}^{\prime}(1/2)=0$. Expanding $\psi_{r}^{\prime}(1/2)$ gives $2g_{w}(1/2)^{r-1}(rg_{w}^{\prime}(1/2))=0$. Since $g_{w}(1/2)=\hat{w}(\emptyset)^{2}>0$, we thus require

[TABLE]

In order to satisfy (13), we need $\hat{w}(S)=0$ for all $S\subseteq[k]$ where $|S|=1$. To use Lemma 5, we would like to choose $w$ such that (13) holds. We discuss how to choose $w$ to optimize our lower bounds in the appendix. In the next section, we will provide $r$ so that the conditions of Lemma 5 hold at $\alpha=1/2$ for arbitrary $w$ when (13) is satisfied.

4.1 Bounding the Second Moment For Fixed $w$

We give a general bound on $r$ in terms of our weight function $w$ so that the conditions of Lemma 5 are satisfied for $\alpha=1/2$. For now, the only constraint we place on $w$ is that (13) holds. The next lemma lets us consider only $\alpha\in[1/2,1]$.

Lemma 6**.**

Let $\alpha\geq 1/2$. Then $g_{w}(\alpha)\geq g_{w}(1-\alpha)$.

This lemma follows because $g_{w}(\alpha)$ is a polynomial in $(2\alpha-1)$ with nonnegative coefficients, and $(2\alpha-1)>0$ for $\alpha>1/2$.

Now we can bound $\psi_{r}(\alpha)$ for $\alpha\in[1/2,1]$. Combined with Lemma 6, the next lemma will give conditions on $r$ such that $\psi_{r}(1/2)>\psi_{r}(\alpha)$ for all $\alpha\in[0,1]$.

Lemma 7**.**

Let the weight function $w$ satisfy (13). If

[TABLE]

$\psi_{r}(1/2)>\psi_{r}(\alpha)$* for $\alpha\in[0,1]$ and $\psi_{r}^{\prime\prime}(1/2)<0$.*

Figure 2c shows how $r$ controls the shape of the function $\psi_{r}(\alpha)$. As $r$ increases, $\psi_{r}^{\prime\prime}(1/2)$ becomes positive and $\psi_{r}(\alpha)$ will no longer attain a local maximum in that region. The key step in proving Lemma 7 is rearranging $\psi_{r}(1/2)>\psi_{r}(\alpha)$ and simplify calculations by using approximations for the logarithmic terms that appear.

Our bound on $r$ compares the average of $w$ over $\{-1,1\}^{k}$ with the correlations between $w$ and the Fourier basis functions. If $w$ has strong correlations with the other Fourier basis functions, two assignments which are equal at $\alpha n$ variables will likely either be both satisfying or both not satisfying as $\alpha$ approaches $1$. This increases $\text{E}[X^{2}]$ but not $\text{E}[X]^{2}$ and makes Lemma 2 provide a trivial bound if $r$ is too large. Thus, if $w$ has strong correlations with the Fourier basis functions, we must choose smaller $r$ as reflected by (14).

To get the tightest bounds, we wish to maximize the expression in (14). Although we prove our lemma for general $w$ requiring only (13), we also need $w(u)=0$ whenever $f(u)=0$ to apply our lemma to satisfiability. Recalling our definition of $X$ in (5), this condition ensures that $C_{f}(n,rn)$ has a solution whenever $X\neq 0$. Thus,

[TABLE]

for some $\lambda:\{-1,1\}^{k}\rightarrow\mathbb{R}$. If we disregard (13), choosing $\lambda(u)=1$ would maximize the bound on $r$ in (14). The additional requirement of (13) for the second moment method to succeed can be viewed as a “symmetrization penalty” on $r$. In the appendix, we discuss how to choose $w$ to optimize our bound on $r$ while satisfying (13) and (15).

4.2 Proving the Main Theorem

We will combine our lemmas to prove Theorem 1.

Proof of Theorem 1.

We wish to apply the second moment method on $X$ defined in (5), where $w$ is a function we use to weigh assignments to individual constraints. We choose $w$ as described in the full version of the paper, which satisfies both (13) and (15). Since $w$ satisfies (15), $\Pr[C_{f}(n,rn)\ \text{is satisfiable}]\geq\Pr[X\neq 0]\geq\text{E}[X]^{2}/\text{E}[X^{2}]$ by Lemma 2. Now we will use Lemma 7 to show that the conditions for Lemma 5 are satisfied for $\phi_{r}=g_{w}(\alpha)^{r}$ and $r$ satisfying (14). For our choice of $w$, it follows from the derivations in the appendix that the RHS of (14) becomes

[TABLE]

where $A$ is defined in Section 3. There is a slight technicality in directly applying Lemma 5 because $\phi_{r}$ might not be nonnegative for $\alpha<1/2$; we discuss this in the appendix. Now using (11) and (12), and applying Lemma 5, we can conclude that there exists $C>0$ such that

[TABLE]

for sufficiently large $n$ and all $r<r_{f,\text{low}}$. ∎

There remains a question of what $r_{f,\text{low}}$ we can hope achieve using a second moment method proof where $X$ is defined as in (5). The following lemma provides some intuition for this:

Lemma 8**.**

In order for the conditions of Lemma 5 to hold at $\alpha_{\max}=1/2$ for $X$ in the form of (5) and any choice of $w$ satisfying (15), we require

[TABLE]

The difference of $\boldsymbol{1}^{T}A^{+}A\boldsymbol{1}/2^{k}$ in the lower logarithm compared to $r_{f,\text{up}}$ can be viewed as a “symmetrization penalty” necessary for our proof to work. While Lemma 8 does not preclude applications of the second moment method that do not rely on Lemma 5, consider what happens for $r$ that do not satisfy (16). For these $r$, the function $\psi_{r}(\alpha)$ must obtain a maximum at some $\alpha^{*}\in[0,1],\alpha^{*}\neq 1/2$. If it also happens that $\phi_{r}(\alpha)$ is nonnegative and twice differentiable on $[0,1]$, and $\psi_{r}^{\prime\prime}(\alpha^{*})<0$, then conditions of Lemma 5 hold, and applying it to $\alpha_{\max}=\alpha^{*}$ along with (10), (11), and (12) will actually imply that

[TABLE]

for some constant $B>0$, which gives us an exponentially decreasing, and therefore trivial lower bound for the second moment method. Therefore, we believe that (16) is near the best lower bound on $r_{f,\text{unsat}}$ that we can achieve by applying the second moment method on $X$ in the form of (5).

5 Experimental Verification of Bounds

We empirically test our bounds with the goal of examining their tightness. For our constraint functions, we will use tribes functions. The tribes function takes the disjunction of $b$ groups of $a$ variables and evaluates to 1 or 0 based on whether the following formula is true:

[TABLE]

where $+1$ denotes true and $-1$ denotes false. For our experiments, we randomly generate CSP formulas based on $\text{TRIBES}_{a,b}$. We use the Dimetheus111https://www.gableske.net/dimetheus random CSP solver to solve these formulas, or report if no solution exists. We show our results in Figure 3. As expected, our values for lower bounds $r_{f,\text{low}}$ are looser than the upper bounds $r_{f,\text{up}}$.

6 Conclusion

Using Fourier analysis and the second moment method, we have shown general bounds on $m/n$, the ratio of constraints to variables; for $m/n$ below these bounds, there is constant probability that a random CSP is satisfiable. We demonstrate that our bounds are easily instantiated and can be applied to obtain novel estimates of the satisfiability threshold for many classes of CSPs. Our bounds depend on how easy it is to symmetrize solutions to the constraint function. We provide a heuristic argument to approximate the best possible lower bounds that our application of the second moment method can achieve; these bounds differ from upper bounds on the satisfiability threshold by a “symmetrization penalty.” Thus, an interesting direction of future research is to determine whether we can provide tighter upper bounds that account for symmetrization, or whether symmetrization terms are an artificial product of the second moment method.

7 Acknowledgements

This work was supported by the Future of Life Institute (grant 2016-158687) and by the National Science Foundation (grant 1649208).

8 Appendix: Optimizing $w$

Recall that we wish to choose $w$, our function used to weight solutions to the constraints in $C_{f}(n,rn)$ so that $\hat{w}(\emptyset)$ is as close as possible to $\hat{f}(\emptyset)$, where $f$ is our constraint function. We require that (13) and (15) hold, and we wish to maximize the bound in Lemma 7. For convenience, we let $\hat{w}=(\hat{w}(S))_{S\subseteq[k]}$, the vector of $2^{k}$ Fourier coefficients of $w$. The bound in (14) depends only on $\hat{w}(\emptyset)^{2}/(\|\hat{w}\|_{2}^{2}-\hat{w}(\emptyset)^{2})$, so we can write the following optimization problem:

[TABLE]

The combination of the objective and the first constraint ensures that we maximize $\hat{w}(\emptyset)^{2}/(\|\hat{w}\|_{2}^{2}-\hat{w}(\emptyset)^{2})$. To see this, we note that we can scale $w$ so $\|w\|_{2}^{2}=1$ without changing the value of $\hat{w}(\emptyset)^{2}/(\|\hat{w}\|_{2}^{2}-\hat{w}(\emptyset)^{2})$. The second constraint ensures that (13) holds, and the third constraint is (15). In this section, we will derive the solution to this optimization problem.

Recalling that (15) asserts that $w(u)=\lambda(u)f(u)$, we will express the Fourier coefficients of $w$ in terms of $\lambda$ and $f$. For convenience, we will let $U=\{u\in\{-1,1\}^{k}:f(u)=1\}$. We note that from (3), for any subset $S\subseteq[k]$

[TABLE]

We formulate this as a matrix-vector product. Overloading notation, we define the $2^{k}$-dimensional vector $\chi(u)=(\chi_{S}(u))_{S\subseteq[k]}$. Define the $2^{k}\times|U|$ matrix $M$ whose columns are the vectors $\chi(u)$ for all $u\in U$, i.e. the matrix whose columns are the Fourier basis functions evaluated at solutions of $f$. Finally, we let $\lambda$ be the $|U|$-dimensional vector whose entries correspond to $\lambda(u)$, arranged in the same order as the columns of $M$. Then from (18), $M\lambda=2^{k}\hat{w}$. Using this, we will transform the optimization problem in (17) into a problem over $\lambda$.

First, the condition that $\hat{w}_{S:|S|=1}=0$ becomes the condition that $A\lambda=0$, where $A=M_{S:|S|=1}$, the submatrix of $M$ consisting of all rows corresponding to size 1 subsets of $[k]$. Let $A^{+}$ denote the Moore-Penrose pseudoinverse of $A$. Since $\lambda$ is in the null space of $A$, we can write it as the projection of any arbitrary vector $\beta\in\mathbb{R}^{|U|}$ as follows:

[TABLE]

where $I-A^{+}A$ is a symmetric, idempotent projection matrix (?). The following example illustrates our notation.

Example 5** ($3$-MAJORITY).**

In the case of $3$-MAJORITY, we can instantiate $M$ as follows, up to permutation of the columns. The submatrix $A$ consists of the second to fourth rows of $M$. We label rows by which basis function they correspond to.

[TABLE]

Now we opt to write our constraint and objective in terms of $\beta$. We note that

[TABLE]

because the columns of $M$ are orthogonal to each other. Plugging in our expression from (19) and using the fact that $I-A^{+}A$ is symmetric and idempotent, we get

[TABLE]

which gives us the transformed constraint

[TABLE]

Next, we transform our objective. Since $\hat{w}(\emptyset)=\boldsymbol{1}^{T}\lambda/2^{k}$, plugging in (19) and including the constraint (20), we get the final optimization problem

[TABLE]

The following lemma gives us the optimal solution to this problem:

Lemma 9**.**

The (not necessarily unique) optimal $\beta$ for this optimization problem occurs at

[TABLE]

for which the corresponding value of $\hat{w}(\emptyset)$ is

[TABLE]

Proof.

Since $I-A^{+}A$ is idempotent and symmetric, an equivalent optimization problem as (21) is

[TABLE]

From this, it is clear that the direction of $(I-A^{+}A)\beta$ which maximizes the objective occurs when $(I-A^{+}A)\beta=\rho(I-A^{+}A)\boldsymbol{1}$, in which case we can choose $\beta=\rho\boldsymbol{1}$ for some $\rho\in\mathbb{R}$. Solving for $\rho$ gives us the desired result. ∎

If we plug this value for $\hat{w}(\emptyset)$ into the bound in Lemma 7, we will get the value of $r_{f,\text{low}}$ described in Theorem 1.

Lemma 9 does not only give us the maximum value of the bound in Lemma 7; it also gives us a sense of the best $r_{f,\text{low}}$ we can achieve using our proof techniques. We can use it to prove Lemma 8:

Proof of Lemma 8.

First, recall that in order for Lemma 5 to hold at $\alpha_{\max}=1/2$, we require (13) to be true. Now for any $w$, we also require $\psi_{r}(1/2)>\psi_{r}(1)$ if the lemma is to apply for $\alpha_{\max}=1/2$. Expanding this condition, we get

[TABLE]

Since $g_{w}(1)/g_{w}(1/2)>1$, the logarithm is positive so we can divide both sides and expand to get

[TABLE]

We obtained the last line by maximizing the RHS over all $w$ that satisfy (13) and (15) and noting that we can apply our solution from Lemma 9 here. ∎

9 Appendix: Proofs and Loose Ends

9.1 Proofs

Proof of Lemma 1.

In the case where the rows of $A$ are linearly independent, a property of the pseudoinverse tells us that $A^{+}=A^{T}(AA^{T})^{-1}$ (?). Then by noting that $A\boldsymbol{1}=2^{k}\hat{f}_{S:|S|=1}$ and $AA^{T}=2^{k}B$,

[TABLE]

∎

Proof of Lemma 4.

As before, we will expand $\text{E}[X^{2}]$ as follows:

[TABLE]

Now we can compute $\text{E}[w(\sigma_{I,s})\tau(\sigma_{I,s})]$ based on the number of indices $i$ where $\sigma_{i}=\tau_{i}$ which we will denote as $\alpha(\sigma,\tau)n$. First, we claim that for any $u,v\in\{-1,1\}^{k}$,

[TABLE]

This holds since we can compute (23) by first computing the probability that $\sigma_{i_{j}}\tau_{i_{j}}=u_{j}v_{j}$ for all $i_{j}\in I$ and then the probability that $s_{j}\sigma_{i_{j}}$ and $s_{j}\tau_{i_{j}}$ actually match $u_{j}$ and $v_{j}$ given that the former is true. We can compute the probability of the former by noting that if $u_{j}v_{j}=1$, then $i_{j}$ must correspond to one of the $\alpha(\sigma,\tau)$ indices where $\sigma_{i_{j}}=\tau_{i_{j}}$. The probability of the latter is simply $1/2^{k}$ since $s$ is chosen uniformly from $\{-1,1\}^{k}$. Now note that

[TABLE]

Expanding the probability using (23) and writing $w$ in terms of its Fourier expansion, we get

[TABLE]

We can factorize the inner part of the summation as follows:

[TABLE]

Thus, terms where $S\neq T$ cancel to 0, so plugging the result back into (24) gives

[TABLE]

Plugging this expression back into (22), we get that

[TABLE]

Now for every $\sigma$ and $j=0,\ldots,n$, there are exactly ${n\choose j}$ choices of $\tau$ for which $\alpha(\sigma,\tau)=j/n$. Thus, combining terms this way finally gives us the expression in (8). ∎

Proof of Lemma 6.

We have

[TABLE]

Since $2\alpha-1\geq 0$ for $\alpha\geq 1/2$, all terms in the sum will be positive so $g_{w}(\alpha)\geq g_{w}(1-\alpha)$. ∎

Proof of Lemma 7.

We first show that $\psi_{r}(\alpha)$ takes its maximum at $\alpha=1/2$. For all $\alpha\in[1/2,1]$, we wish to prove

[TABLE]

Rearranging and taking the logarithm of both sides gives

[TABLE]

Since $g_{w}(\alpha)>\hat{w}(\emptyset)^{2}=g_{w}(1/2)$ for $\alpha>1/2$, $\log\frac{g_{w}(\alpha)}{g_{w}(1/2)}$ is nonnegative so we can divide it from both sides. Thus, it suffices to have

[TABLE]

By using the rule $\log(1+x)\leq x$, we obtain

[TABLE]

We got to the second line by expanding and applying the fact that (13) holds. Plugging this into (25) shows that it is sufficient for

[TABLE]

We can check that the function

[TABLE]

is increasing on the interval $[1/2,1]$, and in addition $\lim_{\alpha\rightarrow 1/2}h(\alpha)=1/2$. Meanwhile,

[TABLE]

at $\alpha=1$. Therefore, for all $\alpha\in[1/2,1]$, the RHS of (26) is lower bounded by

[TABLE]

which is precisely our condition on $r$ in (14). Thus, if (14) holds, then (26) must hold for $\alpha\in[1/2,1]$, which implies that $\psi_{r}(\alpha)$ is maximized at $\alpha=1/2$. Now we check that $\psi_{r}^{\prime\prime}(1/2)<0$. We can first compute the first derivative as

[TABLE]

Since $g_{w}^{\prime}(1/2)=0$ from (13), $\psi^{\prime\prime}(1/2)<0$ if the derivative of

[TABLE]

is negative. We can compute this derivative as

[TABLE]

Plugging in $\alpha=1/2$, the expression simplifies to

[TABLE]

Thus, if (14) holds,

[TABLE]

which means that $\psi_{r}^{\prime\prime}(1/2)$ is also satisfied. ∎

Using the lemmas developed earlier regarding the moments of $X$, we can complete the proof of Proposition 1.

Proof of Proposition 1.

We let $X$ be the number of solutions to $C_{f}(n,rn)$. This corresponds to the choice of $w=f$. Then Lemma 3 tells us that

[TABLE]

With

[TABLE]

we get $2\hat{f}(\emptyset)^{r}<1$. Therefore, $\lim_{n\rightarrow\infty}\text{E}[X]=0$. By Markov’s inequality,

[TABLE]

as desired. ∎

Next, we justify our application of Lemma 5 from Section 4.2 in greater detail.

Detailed application of Lemma 5.

We see that we cannot apply Lemma 5 directly to $\phi_{r}(\alpha)=g_{w}(\alpha)^{r}$, as $\phi_{r}(\alpha)$ may be negative for $\alpha<1/2$ if $w$ can take on negative values. Instead, we define

[TABLE]

The idea is to apply Lemma 5 to $\phi_{r}^{*}(\alpha)$ instead. The benefit of doing this is that $\phi_{r}^{*}(\alpha)>0$ for all $\alpha\in[0,1]$, as $g_{w}(\alpha)>0$ if $\alpha\geq 1/2$. We will also check that $\phi_{r}^{*}(\alpha)$ is twice differentiable on $[0,1]$. For $\alpha\neq 1/2$, this is clear. Now we note that

[TABLE]

At $\alpha=1/2$, this evaluates to [math] because of (13). Likewise, the derivative of $g_{w}(\alpha)^{r}$ at $\alpha=1/2$ is also [math]. Thus, $\phi_{r}^{*}(\alpha)$ is first-order differentiable on $[0,1]$. To show that the second derivative exists at $\alpha=1/2$, we note that

[TABLE]

If we compute the same expression for the second derivative of $g_{w}(\alpha)^{r}$, we see that the two expressions have identical terms at $\alpha=1/2$. Thus, second derivatives match, so $\phi_{r}^{*}(\alpha)$ is twice-differentiable on $[0,1]$.

Now we can apply Lemma 5 to $\phi_{r}^{*}$. Define

[TABLE]

Since $\alpha^{\alpha}(1-\alpha)^{1-\alpha}$ is symmetric around $1/2$, Lemma 7 implies that for $r$ satisfying (14), $\psi_{r}^{*}(1/2)>\psi_{r}^{*}(\alpha)$ for all $\alpha\in[0,1]$ where $\alpha\neq 1/2$. Furthermore, since $(\psi_{r}^{*})^{\prime\prime}(1/2)=\psi_{r}^{\prime\prime}(1/2)<0$ by symmetry around $1/2$, the conditions for Lemma 5 are satisfied so

[TABLE]

for some constant $C>0$ and sufficiently large $n$. Finally, we note that from (8),

[TABLE]

because $\phi_{r}(\alpha)\leq\phi_{r}^{*}(\alpha)$ for $\alpha<1/2$, as a consequence of Lemma 6. Plugging this result into (27) finally gives us

[TABLE]

∎

9.2 Sampling With vs. Without Replacement

We will provide an explanation for why we can assume that we can sample constraints and constraint indices with replacement when proving our main theorem. This explanation is due to Section 3 of (?) for $k$-SAT, and applies directly to our more general setting.

We call a constraint $f_{I,s}$ improper if $I$ contains repeated variables or $f_{I,s}$ is itself repeated. The probability that a constraint contains repeated variables is bounded above by $k^{2}/n$, so with high probability there are $o(n)$ constraints with repeated variables. Likewise, with high probability there are $o(n)$ repeated clauses, so there are $o(n)$ improper constraints with high probability. Furthermore, the distribution over proper constraints remains uniform in the with-replacement setting. Thus, if $C_{f}(n,rn)$ is satisfiable with constant probability in the with-replacement setting for $m=rn$, then it will be satisfiable with constant probability in the without-replacement setting for $m=rn-o(n)$ constraints. Since we only subtract a $o(n)$ factor, we retain the same ratio $r$.