Generalization Bounds for Set-to-Set Matching with Negative Sampling

TL;DR

This paper provides a theoretical analysis of the generalization error in set-to-set matching tasks using neural networks, addressing a gap in understanding the behavior of such models.

Contribution

It introduces a novel generalization bound for set-to-set matching with neural networks, incorporating negative sampling techniques.

Findings

Derived a new generalization bound for set-to-set matching models.

Analyzed the impact of negative sampling on model generalization.

Provides insights into the theoretical behavior of neural set matching.

Abstract

The problem of matching two sets of multiple elements, namely set-to-set matching, has received a great deal of attention in recent years. In particular, it has been reported that good experimental results can be obtained by preparing a neural network as a matching function, especially in complex cases where, for example, each element of the set is an image. However, theoretical analysis of set-to-set matching with such black-box functions is lacking. This paper aims to perform a generalization error analysis in set-to-set matching to reveal the behavior of the model in that task.

Equations

$$f(\mathcal{X},\mathcal{Y})=f({\pi }_x\mathcal{X},{\pi }_y\mathcal{Y})$$

$$f({\pi }_x\mathcal{X},{\pi }_y\mathcal{Y})={\pi }_{x}f(\mathcal{X},\mathcal{Y})$$

$$f(\mathcal{X},\mathcal{Y})=f(\mathcal{Y},\mathcal{X}).$$

$$pf({\mathcal{Z}}^{(1)},{\mathcal{Z}}^{(2)})=f({\mathcal{Z}}^{(p(1))},{\mathcal{Z}}^{(p(2))})$$

$$\ell (f,Z^{+},Z^{-}):=\phi (f(Z^{+})-f(Z^{-})),$$

$$\phi (f(Z^{+})-f(Z^{-}))=\log \left\{1+\exp (-(f(Z^{+})-f(Z^{-}))\right\}.$$

$$R(f)$$

$$\hat{R}(f;S):=\frac{1}{m^{+}{m}^{-}}\sum _{i=1}^{m^{+}}\sum _{j=m^{+}+1}^{m^{+}+m^{-}}\ell (f,Z^{+},Z^{-}).$$

$$\left|\phi (a)-\phi (b)\right|\le L\cdot \left|a-b\right|,$$

$${\hat{\mathcal{R}}}_S(\mathcal{F}):={\mathbb{E}}_{\sigma }\left[\underset{f\in \mathcal{F}}{\sup }\frac{1}{m}\sum _{i=1}^m{\sigma }_{i}f(Z_i)\right].$$

$${\mathcal{R}}_m(\mathcal{F}):={\mathbb{E}}_{S\sim p^m}\left[\hat{\mathcal{R}}(\mathcal{F})\right].$$

$${\varphi }_{\rho }(z)=\{\begin{array}{ll}0& (\rho \le z)\\ 1-z/\rho & (0\le z\le \rho )\\ 1& (z\le 0).\end{array}$$

$$\mathbb{E}[g(z)]$$

$$\psi (S^{\prime })-\psi (S)\le \underset{g\in \mathcal{G}}{\sup }\frac{g(z_m)-g(z_m^{\prime })}{m}\le \frac{1}{m}.$$

$$\psi (S)\le {\mathbb{E}}_S[\psi (S)]+\sqrt{\frac{\log \frac{2}{\delta }}{2m}}.$$

$${\mathbb{E}}_S[\psi (S)]$$

$$\le {\mathbb{E}}_{S,S^{\prime }}\left[\underset{g\in \mathcal{G}}{\sup }\frac{1}{m}\sum _{i=1}^m(g(z_i^{\prime })-g(z_i))\right]$$

$$={\mathbb{E}}_{\sigma ,S,S^{\prime }}\left[\underset{g\in \mathcal{G}}{\sup }\frac{1}{m}\sum _{i=1}^m{\sigma }_i(g(z_i^{\prime })-g(z_i))\right]$$

$$\le {\mathbb{E}}_{\sigma ,S^{\prime }}\left[\underset{g\in \mathcal{G}}{\sup }\frac{1}{m}\sum _{i=1}^m{\sigma }_{i}g(z_i^{\prime })\right]+{\mathbb{E}}_{\sigma ,S}\left[\underset{g\in \mathcal{G}}{\sup }\frac{1}{m}\sum _{i=1}^m-{\sigma }_{i}g(z_i)\right]$$

$$=2{\mathbb{E}}_{\sigma ,S}\left[\underset{g\in \mathcal{G}}{\sup }\frac{1}{m}\sum _{i=1}^m{\sigma }_{i}g(z_i)\right]=2{\mathcal{R}}_m(\mathcal{G}).$$

$${\mathcal{R}}_m(\mathcal{G})\le {\hat{\mathcal{R}}}_S(\mathcal{G})+\sqrt{\frac{\log \frac{2}{\delta }}{2m}}.$$

$$\varphi (S)\le 2{\hat{\mathcal{R}}}_S(\mathcal{G})+3\sqrt{\frac{\log \frac{2}{\delta }}{2m}}.$$

$$R(f)$$

$$\mathbb{E}\left[{\varphi }_{\rho }(a[f(Z^{+})-f(Z^{-})])\right]\le {\hat{R}}_{\rho }(f)+2{\mathcal{R}}_m({\varphi }_{\rho }\circ \tilde{\mathcal{F}})+\sqrt{\frac{\log \frac{1}{\delta }}{2m}}.$$

$$R(f)\le {\hat{R}}_{\rho }(f)+2{\mathcal{R}}_m({\varphi }_{\rho }\circ \tilde{\mathcal{F}})+\sqrt{\frac{\log \frac{1}{\delta }}{2m}}.$$

$${\mathcal{R}}_m(\tilde{\mathcal{F}})$$

$$=\frac{1}{m}{\mathbb{E}}_{S,\sigma }\left[\underset{f\in \mathcal{F}}{\sup }\sum _{i=1}^m{\sigma }_i(f(Z_i^{\prime })-f(Z_i))\right]$$

$$\le \frac{1}{m}{\mathbb{E}}_{S,\sigma }\left[\underset{f\in \mathcal{F}}{\sup }\sum _{i=1}^m{\sigma }_{i}f(Z_i^{\prime })+\underset{f\in \mathcal{F}}{\sup }\sum _{i=1}^m{\sigma }_{i}f(Z_i)\right]$$

$$={\mathbb{E}}_S\left[{\mathcal{R}}_{S^2}(\mathcal{F})+{\mathcal{R}}_{S^1}(\mathcal{F})\right]={\mathcal{R}}_m^{p_2}(\mathcal{F})+{\mathcal{R}}_m^{p_1}(\mathcal{F}).$$

$${\mathcal{F}}_r=\{f\in {\mathfrak{R}}_K|\Vert f{\Vert }_K\le r\}$$

$${\mathbb{P}}_{S_{\alpha }}\left[|\hat{R}(f;S_{\alpha })-R(f)|\ge ϵ\right]\le 2\exp \left\{\frac{{\alpha }^2(1-\alpha {)}^{2}m{ϵ}^2}{2L^2{\kappa }^2{r}^2}\right\},$$

$$z_i:=z(Z_i):==\{\begin{array}{ll}+1& (Z_i\in S^{+}),\\ -1& (Z_i\in S^{-}).\end{array}$$

$$|\hat{R}(f;S)-\hat{R}(f;S^k)|$$

Full text

1 Introduction

The problem of matching two sets of multiple elements, namely set-to-set matching, has received a great deal of attention in recent years [3, 6, 7, 16]. The problem is formalized as a task that, given two distinct sets, finds the goodness of match between them. In particular, when the elements of the set are high-dimensional, neural networks are used as the matching function [11]. Although these strategies have been reported to work well experimentally, there is a lack of research on their theoretical behavior. A mathematical understanding of the behavior of the algorithm is an important issue since a lack of theoretical research hinders the improvement of existing algorithms for set-to-set matching.

We aim to perform a generalization error analysis of set-to-set matching algorithms in the context of statistical learning theory [15, 14]. In particular, existing deep learning-based set-to-set matching algorithms rely on negative sampling, a procedure in which negative examples are randomly generated while learning process [11]. Therefore, we clarify the theoretical behavior of the set-to-set matching algorithm with negative sampling.

2 Preliminaries

Let $\bm{x}_{n},\bm{y}_{m}\in\mathfrak{X}=\mathbb{R}^{d}$ be $d$-dimensional feature vectors representing the features of each individual item. Let $\mathcal{X}=\{\bm{x}_{1},\dots,\bm{x}_{N}\}$ and $\mathcal{Y}=\{\bm{y}_{1},\dots,\bm{y}_{M}\}$ be sets of these feature vectors, where $\mathcal{X},\mathcal{Y}\in 2^{\mathfrak{X}}$ and $N,M\in\mathbb{N}$ are sizes of the sets. The function $f:2^{\mathfrak{X}}\times 2^{\mathfrak{X}}\to\mathbb{R}$ calculates a matching score between the two sets $\mathcal{X}$ and $\mathcal{Y}$. Guaranteeing the exchangeability of the set-to-set matching requires that the matching function $f(\mathcal{X},\mathcal{Y})$ is symmetric and invariant under any permutation of items within each set as follows.

Definition 1 (Permutation Invariance)

A set-input function f is said to be permutation invariant if

[TABLE]

for permutations $\pi_{x}$ on $\{1,\dots,N\}$ and $\pi_{y}$ on $\{1,\dots,M\}$.

Definition 2 (Permutation Equivariance)

A map $f:\mathfrak{X}^{N}\times\mathfrak{X}^{M}\to\mathfrak{X}^{N}$ is said to be permutation equivariant if

[TABLE]

for permutations $\pi_{x}$ and $\pi_{y}$, where $\pi_{x}$ and $\pi_{y}$ are on $\{1,\dots,N\}$ and $\{1,\dots,M\}$, respectively. Note that $f$ is permutation invariant for permutations within $\mathcal{Y}$.

Definition 3 (Symmetric Function)

A map $f:2^{\mathfrak{X}}\times 2^{\mathfrak{X}}\to\mathbb{R}$ is said to be symmetric if

[TABLE]

Definition 4 (Two-Set-Permutation Equivariance)

Given $\mathcal{X}^{(1)}\in\mathfrak{X}^{N}$ and $\mathcal{Z}^{(2)}\in\mathfrak{X}^{M}$, a map $f:\mathfrak{X}^{\ast}\times\mathfrak{X}^{\ast}\to\mathfrak{X}^{\ast}\times\mathfrak{X}^{\ast}$ is said to be two-set-permutation equivariant if

[TABLE]

for any permutation operator $p$ exchanging the two sets, where $\mathfrak{X}^{\ast}=\cup^{\infty}_{n=0}\mathfrak{X}^{n}$ indicates a sequence of arbitrary length such as $\mathfrak{{X}^{N}}$ or $\mathfrak{X}^{M}$.

We consider tasks where the matching function f is used per pair of sets [18] to select a correct matching. Given candidate pairs of sets $(\mathcal{X},\mathcal{Y}^{(k)})$, where $\mathcal{X},\mathcal{Y}^{(k)}\in 2^{\mathfrak{X}}$ and $k\in\{1,\dots,K\}$, we choose $\mathcal{Y}^{(k^{\ast})}$ as a correct one so that $f(\mathcal{X},\mathcal{Y}^{(k^{\ast})})$ achieves the maximum score from amongst the $K$ candidates.

2.1 Set-to-set matching with negative sampling

In real-world set-to-set matching problems, it is often the case that only positive example set pairs can be obtained. Then, we consider training a model for set-to-set matching with negative sampling. The learner is given positive examples $S^{+}=\{(\mathcal{X},\mathcal{Y})\}^{m^{+}}_{i=1}$. Then, negative examples $S^{-}=\{(\mathcal{X},\mathcal{Y})\}^{m^{-}}_{i=1}$ are generated by randomly combining set pairs from the given sets. We assume that positive and negative examples are drawn according to the underlying distribution $p^{+}$ and $p^{-}$, respectively. Given training sample set $S=(S^{+},S^{-})$, the goal of set-to-set matching with negative sampling is to learn a real-valued score function $f:2^{\mathfrak{X}}\times 2^{\mathfrak{X}}\to\mathbb{R}$ that ranks future positive pair $(\mathcal{X},\mathcal{Y})^{+}$ higher than negative pair $(\mathcal{X},\mathcal{Y})^{-}$. Let $\ell$ be the loss function, which is defined as

[TABLE]

where $Z^{+}=(\mathcal{X},\mathcal{Y})^{+}$, $Z^{-}=(\mathcal{X},\mathcal{Y})^{-}$ and $\varphi:\mathbb{R}\to\mathbb{R}^{+}$ is a convex function. Typical choices of $\varphi$ include the logistic loss

[TABLE]

Definition 5 (Expected set-to-set matching loss)

Expected set-to-set matching loss $R(f)$ is defined as

[TABLE]

Definition 6 (Empirical set-to-set matching loss)

Empirical set-to-set matching loss $\hat{R}(f;S)$ is defined as

[TABLE]

Here, we assume that $\varphi$ has the Lipschitz property with respect to $\mathbb{R}$, i.e.,

[TABLE]

where $a,b\in\mathbb{R}$ and $L>0$ is a Lipschitz constant.

3 Margin bound for set-to-set matching

Our first result is based on the Rademacher complexity.

Definition 7 (Empirical Rademacher complexity)

Let $\mathcal{F}$ be a family of matching score functions. Then, the empirical Rademacher complexity of $\mathcal{F}$ with respect to the sample $S$ is defined as

[TABLE]

Definition 8 (Rademacher complexity)

Let $p$ denote the distribution according to which samples are drawn. For any integer $m\geq 1$, the Rademacher complexity of $\mathcal{F}$ is the expectation of the empirical Rademacher complexity over all samples of size $m$ drawn according to $p$:

[TABLE]

Let $p_{1}$ the marginal distribution of the first element of the pairs, and by $p_{2}$ the marginal distribution with respect to the second element of the pairs. Similarly, $S^{1}\sim p_{1}$ and $S^{2}\sim p_{2}$. We denote by $\mathcal{R}^{1}_{m}$ the Rademacher complexity of $\mathcal{F}$ with respect to the marginal distribution $p_{1}$, that is $\mathcal{R}^{1}_{m}(\mathcal{F})=\mathbb{E}[\hat{\mathcal{R}}_{S^{1}}(\mathcal{F})]$, and similarly $\mathcal{R}^{2}_{m}(\mathcal{F})=\mathbb{E}[\hat{\mathcal{R}}_{S^{2}}(\mathcal{F})]$.

Here, we assume that the loss function is the following margin loss.

Definition 9

For any $\rho>0$, the $\rho$-margin loss is the function $\ell_{\rho}$ defined for all $z,z^{\prime}\in\mathbb{R}$ by $\ell_{\rho}(z,z^{\prime})=\phi(zz^{\prime})$ with,

[TABLE]

Lemma 1

Let $Z\in\mathbb{R}$ be any input space, and $\mathcal{G}$ be a family of functions mapping from $Z$ to $[0,1]$. Then, for any $\delta>0$, with probability at least $1-\delta$, each of the following holds for all $g\in\mathcal{G}$:

[TABLE]

Proof

Let $\psi(S)=\sup_{g\in\mathcal{G}}\mathbb{E}[g]-\frac{1}{m}\sum^{m}_{i=1}g(z_{i})$. Then, for two samples $S$ and $S^{\prime}$, we have

[TABLE]

where $z_{m}\in S$ and $z^{\prime}_{m}\in S^{\prime}$. Then, by McDiarmid’s inequality, for any $\delta>0$, with probability at least $1-\delta/2$, the following holds.

[TABLE]

We next bound the expectation of the right-hand side as follows.

[TABLE]

Here, using again McDiarmid’s inequality, with probability at least $1-\delta/2$, the following holds.

[TABLE]

Finally, we use the union bound which yields with probability at least $1-\delta$:

[TABLE]

Theorem 3.1 (Margin bound for set-to-set matching)

Let $\mathcal{F}$ be a set of matching score functions. Fix $\rho>0$. Then, for any $\delta>0$, with probability at least $1-\delta$ over the choice of a sample $S$ of size $m$, each of the following holds for all $f\in\mathcal{F}$:

[TABLE]

Proof

Let $\tilde{\mathcal{F}}$ be the family of functions mapping $(\mathfrak{X}\times\mathfrak{X})\times\{-1,+1\}$ to $\mathbb{R}$ defined by $\tilde{\mathcal{F}}=\{z=(Z^{\prime},Z),a)\mapsto a[f(Z^{\prime})-f(Z)]\ |\ f\in\mathcal{F}\}$, where $a\in\{0,1\}$. Consider the family of functions $\tilde{\mathcal{F}}=\{\phi_{\rho}\circ g\ |\ f\in\tilde{\mathcal{F}}\}$ derived from $\tilde{\mathcal{F}}$ which are taking values in $[0,1]$. By Lemma 1, for any $\delta>0$ with probability at least $1-\delta$, for all $f\in\mathcal{F}$,

[TABLE]

Since $1_{u\leq 0}\leq\phi_{\rho}(u)$ for all $u\in\mathbb{R}$, the generalization error $R(f)$ is a lower bound on left-hand side, $R(f)=\mathbb{E}[1_{a[f(Z^{\prime})-f(Z)]\leq 0}]\leq\mathbb{E}[\phi_{\rho}(a[f(Z^{\prime})-f(Z)])]$, and we can write

[TABLE]

Here, we can show that $\mathcal{R}_{m}(\phi_{\rho}\circ\tilde{\mathcal{F}})\leq\frac{1}{\rho}\mathcal{R}_{m}(\tilde{\mathcal{F}})$ using the $(1/\rho)$-Lipschitzness of $\phi_{\rho}$. Then, $\mathcal{R}_{m}(\tilde{\mathcal{F}})$ can be upper bounded as follows:

[TABLE]

4 RKHS bound for set-to-set matching

In this section, we consider more precise bounds that depend on the size of the negative sample produced by negative sampling. Let $S=((\mathcal{X}_{1},\mathcal{Y}_{1}),\dots,(\mathcal{X}_{m},\mathcal{Y}_{m}))\in(\mathfrak{X}\times\mathfrak{X})^{m}$ be a finite sample sequence, and $m^{+}$ be the positive sample size. If the positive proportion $\frac{m^{+}}{m}=\alpha$, then sample sequence $S$ also can be denoted by $S_{\alpha}$.

Let $\mathfrak{R}_{K}$ be the reproducing kernel Hilbert space (RKHS) associated with the kernel $K$, and $\mathcal{F}_{r}$ is defined as

[TABLE]

for $r>0$.

Theorem 4.1 (RKHS bound for set-to-set matching)

Suppose $S_{\alpha}$ to be any sample sequence of size $m$. Then, for any $\epsilon>0$ and $f\in\mathcal{F}_{r}$,

[TABLE]

where $\kappa\coloneqq\sup_{x}\sqrt{K(x,x)}$.

Proof

Denote $S=(S^{+},S^{-})=\{Z_{1},\dots,Z_{m}\}$ and

[TABLE]

First, for each $1\leq k\leq m^{+}$ such that $z_{i}=+1$, let $(Z_{k},+1)$ be replaced by $(Z^{\prime}_{k},+1)\in(\mathfrak{X}\times\mathfrak{X})\times\{-1,+1\}$, and we denote by $S^{k}$ as this sample. Then,

[TABLE]

Next, for each $m^{+}+1\leq k\leq m$ such that $z_{i}=-1$, let $(Z_{l},-1)$ be replaced by $(Z^{\prime}_{k},-1)\in(\mathfrak{X}\times\mathfrak{X})\times\{-1,+1\}$ and we denote by $\bar{S}^{k}$ as this sample. Similarly, we have

[TABLE]

Finally, for each $1\leq k\leq m^{+}$ such that $z_{i}=+1$, let $(Z_{k},+1)$ be replaced by $(Z^{\prime}_{k},-1)\in(\mathfrak{X}\times\mathfrak{X})\times\{-1,+1\}$, and we denote by $\tilde{S}^{k}=\bar{S}^{k}\cup\{(Z_{m+1},-1)\}$ as this sample. Then, we have

[TABLE]

where $\Gamma_{1}=|\hat{R}(f;S)-\hat{R}(f;S\cup\{Z_{m+1},-1\})|$ and $\Gamma_{2}=|\hat{R}(f;S\cup\{Z_{m+1},-1\})-\hat{R}(f;\tilde{S}^{k})|$. Since $\Gamma_{1}\leq\frac{2L}{m^{-}+1}\|f\|_{\infty}$ and $\Gamma_{2}\leq\frac{2L}{m^{+}}\|f\|_{\infty}$, we have

[TABLE]

Combining them and applying McDiarmid’s inequality, we have the proof.

Remark 1

Given $m,\epsilon,L$, we can find that the tight bound can be achieved when $\alpha=\frac{1}{2}$. This means that it is desirable the number of positive samples be equal to the number of negative samples (See Figure 1).

Remark 2

For any $\delta>0$, with probability at least $1-\delta$, we have

[TABLE]

Remark 3

For Remark 2, Let $m=m^{+}+m^{-}$ and fix $m^{+}\in\mathbb{N}$. Then, we have the optimal negative sample size as $(1-\alpha)=2/3$.

5 Conclusion and Discussion

In this paper, we performed a generalization error analysis in set-to-set matching to reveal the behavior of the model in that task. Our analysis reveals what the convergence rate of algorithms in set matching depend on the size of negative sample. Future studies may include the following:

• •

Derivation of tighter bounds. There are many types of mathematical tools for generalization error analysis of machine learning algorithms, and it is known that the tightness of the bounds depends on which one is used. For tighter bounds, it is useful to use mathematical tools not addressed in this paper [1, 9, 8, 10, 2].

• •

Induction of novel set matching algorithms. It is expected to derive a novel algorithm based on the discussion of generalized error analysis.

• •

The effect of data augmentation for generalization error of set-to-set matching. Many data augmentation methods have been proposed to stabilize neural network learning, and theoretical analysis when these are used would be useful [5, 4, 13, 17, 12].