Consistency and Asymptotic Normality of Latent Block Model Estimators

TL;DR

This paper establishes theoretical guarantees for the consistency and asymptotic normality of estimators in the Latent Block Model, extending results to valued data settings and under mild asymptotic conditions.

Contribution

It provides the first theoretical proof of consistency and asymptotic normality for estimators in valued Latent Block Models, including variational estimators.

Findings

MLE is consistent with known labels under mild conditions.

Log-likelihood ratios are equivalent between complete and observed models.

Variational estimators are also consistent and asymptotically normal.

Abstract

The Latent Block Model (LBM) is a model-based method to cluster simultaneously the $d$ columns and $n$ rows of a data matrix. Parameter estimation in LBM is a difficult and multifaceted problem. Although various estimation strategies have been proposed and are now well understood empirically, theoretical guarantees about their asymptotic behavior is rather sparse and most results are limited to the binary setting. We prove here theoretical guarantees in the valued settings. We show that under some mild conditions on the parameter space, and in an asymptotic regime where $\log(d)/n$ and $\log(n)/d$ tend to $0$ when $n$ and $d$ tend to infinity, (1) the maximum-likelihood estimate of the complete model (with known labels) is consistent and (2) the log-likelihood ratios are equivalent under the complete and observed (with unknown labels) models. This equivalence allows us to transfer the asymptotic consistency, and under mild conditions, asymptotic normality, to the maximum likelihood estimate under the observed model. Moreover, the variational estimator is also consistent and, under the same conditions, asymptotically normal.

Equations

$${\mathbf{z}}_i\sim \mathcal{M}(1,\mathbf{\pi }=({\pi }_1,\dots ,{\pi }_g))$$

$${\mathbf{w}}_j\sim \mathcal{M}(1,\mathbf{\rho }=({\rho }_1,\dots ,{\rho }_m))$$

$$X_{ij}|\{z_{ik}{w}_{j\ell }=1\}\sim \phi (.,{\alpha }_{k\ell }).$$

$${IR}_g(\mathbf{z}{)}_{k{k}^{\prime }}=\frac{1}{n}\sum _i{z}_{ik}^{\star }{z}_{i{k}^{\prime }}\text{and}{IR}_m(\mathbf{w}{)}_{\ell {\ell }^{\prime }}=\frac{1}{d}\sum _j{w}_{j\ell }^{\star }{w}_{j{\ell }^{\prime }}$$

$${\mathcal{L}}_c(\mathbf{z},\mathbf{w};\mathbf{\theta })$$

$$=\log \left\{\left(\prod _{i,k}{\pi }_k^{z_{ik}}\right)\left(\prod _{j,\ell }{\rho }_{\ell }^{w_{j\ell }}\right)\left(\prod _{i,j,k,\ell }\phi {\left(x_{ij};{\alpha }_{k\ell }\right)}^{z_{ik}{w}_{j\ell }}\right)\right\}$$

$$=\log \left\{\left(\prod _i{\pi }_{z_i}\right)\left(\prod _j{\rho }_{w_j}\right)\left(\prod _{i,j}\phi \left(x_{ij};{\alpha }_{z_i{w}_j}\right)\right)\right\}.$$

$$\mathcal{L}(\mathbf{\theta })=\log p(\mathbf{x};\mathbf{\theta })=\log \left(\sum _{\mathbf{z}\in \mathcal{Z},\mathbf{w}\in \mathcal{W}}p(\mathbf{x},\mathbf{z},\mathbf{w};\mathbf{\theta })\right).$$

$$\phi (x,\alpha )=b(x)\exp (\alpha x-\psi (\alpha )),$$

$$\mathbb{E}[X_{\alpha }]={\psi }^{\prime }(\alpha )and\mathbb{V}[X_{\alpha }]={\psi }^{\prime \prime }(\alpha ).$$

$$\mathbf{\Theta }\subset [c,1-c{]}^g\times [c,1-c{]}^m\times C_{\alpha }^{g\times m}\text{with}{C}_{\alpha }\subset \mathring{\mathcal{A}}.$$

$$\delta (\mathbf{\alpha })=\min \left\{\underset{\ell \ne {\ell }^{\prime }}{\min }\underset{k}{\max }\mathrm{KL}({\alpha }_{k\ell },{\alpha }_{k{\ell }^{\prime }}),\underset{k\ne k^{\prime }}{\min }\underset{\ell }{\max }\mathrm{KL}({\alpha }_{k\ell },{\alpha }_{k^{\prime }\ell })\right\}$$

$${\mathbf{A}}^s={\left(A_{is(k)}\right)}_{i,k}{\mathbf{B}}^t={\left(B_{jt(\ell )}\right)}_{j,\ell }{\mathbf{C}}^{s,t}={\left(C_{s(k)t(\ell )}\right)}_{k,\ell }$$

$$({\mathbf{\pi }}^s,{\mathbf{\rho }}^t,{\mathbf{\alpha }}^{s,t})=(\mathbf{\pi },\mathbf{\rho },\mathbf{\alpha }).$$

$$\Vert \mathbf{z}-{\mathbf{z}}^{\star }{\Vert }_{0,\sim }=\underset{{\mathbf{z}}^{\prime }\sim \mathbf{z}}{\inf }\Vert {\mathbf{z}}^{\prime }-{\mathbf{z}}^{\star }{\Vert }_0$$

$${\Vert \mathbf{z}\Vert }_0=\sum _{i,k}\text{mathds}1\left\{z_{ik}\ne 0\right\}.$$

$$S({\mathbf{z}}^{\star },{\mathbf{w}}^{\star },r)=\left\{(\mathbf{z},\mathbf{w}):\Vert \mathbf{z}-{\mathbf{z}}^{\star }{\Vert }_{0,\sim }\le rn\text{ and }\Vert \mathbf{w}-{\mathbf{w}}^{\star }{\Vert }_{0,\sim }\le rd\right\}$$

$${\hat{\pi }}_k(\mathbf{z})=\frac{z_{+k}}{n}$$

$${\hat{x}}_{k\ell }(\mathbf{z},\mathbf{w})=\frac{{\sum }_{ij}{x}_{ij}{z}_{ik}{w}_{j\ell }}{z_{+k}{w}_{+\ell }}$$

$$\sqrt{n}\left(\hat{\mathbf{\pi }}\left({\mathbf{z}}^{\star }\right)-{\mathbf{\pi }}^{\star }\right)\underset{n\to \infty }{\overset{\mathcal{D}}{\to }}\mathcal{N}(0,{\Sigma }_{{\mathbf{\pi }}^{\star }})\text{and}\sqrt{d}\left(\hat{\mathbf{\rho }}\left({\mathbf{w}}^{\star }\right)-{\mathbf{\rho }}^{\star }\right)\underset{d\to \infty }{\overset{\mathcal{D}}{\to }}\mathcal{N}(0,{\Sigma }_{{\mathbf{\rho }}^{\star }})$$

$$\sqrt{nd}({\hat{\alpha }}_{k\ell }\left({\mathbf{z}}^{\star },{\mathbf{w}}^{\star }\right)-{\alpha }_{k\ell }^{\star })\underset{n,d\to \infty }{\overset{\mathcal{D}}{\to }}\mathcal{N}(0,{\Sigma }_{{\mathbf{\alpha }}^{\star },k\ell })for\; allk,\ell$$

$${\Sigma }_{{\mathbf{\pi }}^{\star },k{k}^{\prime }}=\{\begin{array}{ll}{\pi }_k^{\star }(1-{\pi }_k^{\star })& \text{if}k=k^{\prime }\\ -{\pi }_k^{\star }{\pi }_{k^{\prime }}^{\star }& \text{if}k\ne k^{\prime }\end{array}\text{and}{\Sigma }_{{\mathbf{\rho }}^{\star },\ell {\ell }^{\prime }}=\{\begin{array}{ll}{\rho }_{\ell }^{\star }(1-{\rho }_{\ell }^{\star })& \text{if}\ell ={\ell }^{\prime }\\ -{\rho }_{\ell }^{\star }{\rho }_{{\ell }^{\prime }}^{\star }& \text{if}\ell \ne {\ell }^{\prime }\end{array}$$

$$\sqrt{nd{\pi }_k^{\star }{\rho }_{\ell }^{\star }}\left({\psi }^{\prime }\left({\hat{\alpha }}_{k\ell }\left({\mathbf{z}}^{\star },{\mathbf{w}}^{\star }\right)\right)-{\psi }^{\prime }({\alpha }_{k\ell }^{\star })\right)$$

$$\underset{n,d\to +\infty }{\overset{\mathcal{D}}{\to }}\mathcal{N}\left(0,{\psi }^{\prime \prime }({\alpha }_{k\ell }^{\star })\right)$$

$$\sqrt{nd}\left({\hat{\alpha }}_{k\ell }\left({\mathbf{z}}^{\star },{\mathbf{w}}^{\star }\right)-{\alpha }_{k\ell }^{\star }\right)\underset{n,d\to +\infty }{\overset{\mathcal{D}}{\to }}\mathcal{N}\left(0,\frac{1}{{\pi }_k^{\star }{\rho }_{\ell }^{\star }{\psi }^{\prime \prime }({\alpha }_{k\ell }^{\star })}\right)$$

$${\mathcal{L}}_c^{\star }\left({\mathbf{\pi }}^{\star }+\frac{s}{\sqrt{n}},{\mathbf{\rho }}^{\star }+\frac{t}{\sqrt{d}},{\mathbf{\alpha }}^{\star }+\frac{u}{\sqrt{nd}}\right)$$

$$-\left(\frac{1}{2}s{}^T{I}_{{\mathbf{\pi }}^{\star }}s+\frac{1}{2}t{}^T{I}_{{\mathbf{\rho }}^{\star }}t+\frac{1}{2}\mathrm{Tr}[(u\odot u){}^T{I}_{{\mathbf{\alpha }}^{\star }}]\right)$$

$$+o_P(1)$$

$${\mathcal{L}}_c^{\star }\left({\mathbf{\pi }}^{\star }+\frac{s}{\sqrt{n}},{\mathbf{\rho }}^{\star }+\frac{t}{\sqrt{d}},{\mathbf{\alpha }}^{\star }+\frac{u}{\sqrt{nd}}\right)$$

$$\underset{k}{\min }{z}_{+k}\ge cn\text{and}\underset{\ell }{\min }{w}_{+\ell }\ge cd.$$

$${\mathbb{P}}_{{\mathbf{\theta }}^{\star }}\left({\bar{\Omega }}_1\right)\le g\exp \left(-\frac{n{c}^2}{2}\right)+m\exp \left(-\frac{d{c}^2}{2}\right).$$

$${\mathbb{P}}_{{\mathbf{\theta }}^{\star }}\left(z_{+k}\le n\frac{c}{2}\right)\le {\mathbb{P}}_{{\mathbf{\theta }}^{\star }}\left(z_{+k}\le n\frac{{\pi }_k}{2}\right)\le \exp \left(-2n{\left(\frac{{\pi }_k}{2}\right)}^2\right)\le \exp \left(-\frac{n{c}^2}{2}\right)$$

$$F_{nd}(\mathbf{\theta },\mathbf{z},\mathbf{w})=\log \frac{p(\mathbf{x}|\mathbf{z},\mathbf{w};\mathbf{\theta })}{p(\mathbf{x}|{\mathbf{z}}^{\star },{\mathbf{w}}^{\star };{\mathbf{\theta }}^{\star })},$$

Full text

Latent Block Model,

asymptotic normality,

Maximum Likelihood Estimate,

Concentration Inequality,

\startlocaldefs\endlocaldefs\arxiv

1704.06629 \startlocaldefs\endlocaldefs

and

and

1 Introduction

Co-clustering is an unsupervised method to cluster simultaneously the $n$ rows and $d$ columns of a rectangular data matrix. The assignments of each row to one of the row-clusters and of each column to one of the column-clusters are unknown and the aim is to determine them. Then, rows and columns can be re-ordered according to their assignments, highlighting the natural structure of the data with distinct blocks having homogeneous observations. This leads to a parsimonious data representation, as can be shown on Figure 1.

Co-clustering can be used in numerous applications, and especially ones with large data sets, such as recommendation systems (to discover a segmentation of customers with regard to a segmentation of products), genomics (to simultaneously define groups of genes having the same expression with regards to groups of experimental conditions) or text mining (to define simultaneously groups of texts and groups of words).

Among the co-clustering methods, the Latent Block Model (LBM) defines a probabilistic model as a mixture model with latent rows and columns assignments. LBM can deal with binary ([6]), Gaussian ([8]), categorical ([9]) or count ([7]) data. Due to the complex dependency structure induced by this modeling, neither the likelihood, nor the distribution of the assignments conditionally to the observations needed in the E-step of the EM algorithm, traditionnally used for mixture models, are numerically tractable. Estimation can be however performed either with a variational approximation leading to an approximate value of the maximum likelihood estimator, or with a Bayesian approach (VBayes algorithm or Gibbs sampler). For example, [9] recommends using a Gibbs sampler combined with a VBayes algorithm.

The asymptotics of the maximum likelihood (MLE) and variational (VE) estimators also raise interesting theoretical questions. This topic was first addressed for the stochastic block model (SBM) ([13]), where the data is a random graph encoded by its adjacency binary matrix: the rows and columns represent the nodes, so that there is only one partition, shared by rows and columns, and a unique asymptotic direction.

For a binary SBM and under the true parameter value, Theorem 3 of [4] states that the distribution of the assignments conditionally to the observations converges to a Dirac of the real assignments. Moreover, this convergence remains valid under the estimated parameter value, assuming that this estimator converges at rate at least $n^{-1}$, where $n$ is the number of nodes (Proposition 3.8). This assumption is not trivial, and it was not established that such an estimator exists except in some particular cases ([1] for example). [10] presented a unified frame for LBM and SBM in case of valued observations satisfying a concentration inequality, and showed the consistency of the conditional distribution of the assignments under all parameter values in a neighborhood of the true value. [3] and [2] proved the consistency and asymptotic normality of the MLE for the binary SBM but failed to account for complications induced by symmetries in the parameter. Building upon the work from [4], they first studied the asymptotic behavior of the MLE in the complete model (observations and assignments) with binary observations which is simple to handle; then, they showed that the complete likelihood and the marginal likelihood have similar asymptotic behaviors by the use of a Bernstein inequality for bounded observations.

Following the main ideas of [2], we prove that the observed likelihood ratio and the complete likelihood ratio computed at the true assignments are asymptotically equivalent, up to a multiplicative term. This term depends on some model symmetry and was omitted in [2] although it is necessary to prove the asymptotic results. We then settle the asymptotic normality of the maximum likelihood and variational estimators. All these results are stated not only for binary observations, but also more generally for observations coming from univariate exponential families in canonical form, which is essential regarding the LBM usages. This leads us to develop a Bernstein-type inequality for sub-exponential variables as the Hoeffing’s concentration inequality used in [2] is only relevant for upper-bounded observations.

The paper is organized as follows. The model, main assumptions and notations are introduced in Section 2, where the concept of model symmetry is also discussed. Section 3 proves the asymptotic normality of the complete likelihood estimator, and section 4 studies conditional and profile log-likelihoods. Our main result showing that the observed likelihood ratio behaves like the complete likelihood ratio is stated in section 5, and its consequences in terms of consistency and asymptotic normality of the MLE and variational estimators are presented in section 6. Most of the proofs are postponed to the appendices to improve the general readibility : appendix A for properties of conditional and profile log-likelihoods, B for the steps of the main result, C for concentration inequalities for specific sub-exponential variables and D for other technical results.

2 Model, assumptions and definitions

We observe a data matrix $X=(x_{ij})$ with $n$ rows and $d$ columns. The LBM assumes that there exists a latent structure in the form of the Cartesian product of a partition of $g$ row-clusters by a partition of $m$ column-clusters with the following characteristics:

• •

the latent row assignments $\mathbf{z}=(\mathbf{z}_{1},\ldots,\mathbf{z}_{n})$ are independent and identically distributed with a common multinomial distribution on $g$ categories:

[TABLE]

For $k=1,\ldots,g$, $z_{ik}=1$ if row $i$ belongs to row-group $k$, [math] otherwise.

In the same way, the latent column assignments $\mathbf{w}=(\mathbf{w}_{1},\ldots,\mathbf{w}_{d})$ are i.i.d. multinomial variables with $m$ categories:

[TABLE]

For $\ell=1,\ldots,m$, $w_{j\ell}=1$ if column $j$ belongs to column-group $\ell$ and [math] otherwise.

• •

the row and column assignments are independent: $p(\mathbf{z},\mathbf{w})=p(\mathbf{z})p(\mathbf{w})$

• •

conditionally to row and column assignments $\mathbf{z}\times\mathbf{w}$, the observed data $X_{ij}$ are independent, and their conditional distribution $\varphi(.,\alpha)$ belongs to the same parametric family, which parameter $\alpha$ only depends on the given block:

[TABLE]

Hence, the complete parameter set is $\boldsymbol{\theta}=(\boldsymbol{\pi},\boldsymbol{\rho},\boldsymbol{\alpha})\in\boldsymbol{\Theta}$, with $\boldsymbol{\alpha}=(\alpha_{11},\dots,\alpha_{gm})$ and $\boldsymbol{\Theta}$ the parameter space. Figure 2 summarizes these notations.

Remark 2.1*.*

Group, class and cluster in one hand, label and assignment in the other hand will be used indistinctly. Moreover, for notation convenience, $\sum_{i}$, $\sum_{j}$, $\sum_{k}$, $\sum_{\ell}$ stand for $\sum_{i=1}^{n}$, $\sum_{j=1}^{d}$, $\sum_{k=1}^{g}$, $\sum_{\ell=1}^{m}$.

When performing inference from data, we denote $\boldsymbol{\theta}^{\star}=(\boldsymbol{\pi}^{\star},\boldsymbol{\rho}^{\star},\boldsymbol{\alpha}^{\star})$ the true parameter set, i.e. the parameter values used to generate the data, and $\mathbf{z}^{\star}$ and $\mathbf{w}^{\star}$ the true (and usually unobserved) row and column assignments. For indicator membership variables $\mathbf{z}$ and $\mathbf{w}$, we also denote:

• •

$z_{+k}=\sum_{i}z_{ik}$ and $w_{+\ell}=\sum_{j}w_{j\ell}$

• •

$z^{\star}_{+k}$ and $w^{\star}_{+\ell}$ their counterpart for $\mathbf{z}^{\star}$ and $\mathbf{w}^{\star}$.

The confusion matrix allows one to compare the partitions.

Definition 2.2** (confusion matrices).**

For given assignments $\mathbf{z}$ and $\mathbf{z}^{\star}$ (resp. $\mathbf{w}$ and $\mathbf{w}^{\star}$), we define the confusion matrix between $\mathbf{z}$ and $\mathbf{z}^{\star}$ (resp. $\mathbf{w}$ and $\mathbf{w}^{\star}$), denoted ${I\!R}_{g}(\mathbf{z})$ (resp. ${I\!R}_{m}(\mathbf{w})$), as follows:

[TABLE]

2.1 Likelihood

When the labels are known, the complete log-likelihood is given by:

[TABLE]

In an unsupervised setting, the labels are unobserved and the observed log-likelihood is obtained by marginalization over all the label configurations:

[TABLE]

Due to the double missing data structure $\mathbf{z}$ for rows and $\mathbf{w}$ for columns, neither the observed likelihood nor the E-step of the EM algorithm are tractable. Estimation can can nevertheless be performed either by numerical approximation, or by MCMC methods ([[, see]]govaert2013co,keribin2015estimation).

2.2 Assumptions

We focus here on LBM where $\varphi$ belongs to a regular univariate exponential family set in canonical form:

[TABLE]

The canonical parameter $\alpha$ belongs to a space $\mathcal{A}$, so that $\varphi(\cdot,\alpha)$ is well defined for all $\alpha\in\mathcal{A}$. Classical properties of exponential families ensure that $\psi$ is convex, infinitely differentiable on $\mathring{\mathcal{A}}$, and $(\psi^{\prime})^{-1}$ is well defined on $\psi^{\prime}(\mathring{\mathcal{A}})$. When $X_{\alpha}\sim\varphi(.,\alpha)$,

[TABLE]

Notice that the definition of the exponential family used here relies on an exhaustive statistic that is $X$ itself. This for a simple convenience. Family sets of the form $\varphi(x,\alpha)=b(x)\exp(\alpha t(x)-\psi(\alpha))$ can also be considered, all the further developments as Bernstein and concentration inequalities then concerning the exhaustive statistics $t(X)$.

Moreover, we make the following assumptions on the parameter space :

1. $H_{1}$

: There exists a positive constant $c$, and a compact $C_{\alpha}$ such that

[TABLE] 2. $H_{2}$

: The true parameter $\boldsymbol{\theta}^{\star}=(\boldsymbol{\pi}^{\star},\boldsymbol{\rho}^{\star},\boldsymbol{\alpha}^{\star})$ lies in the relative interior of $\boldsymbol{\Theta}$. 3. $H_{3}$

: The mixture measure of LBM is identifiable: $\boldsymbol{\theta}^{\star}$ is identifiable up to a permutation of the row-labels and column-labels (see definition 2.7 of equivalent parameters).

The previous assumptions are standard. Notice that the following conditions are necessary for $H_{3}$ to hold:

• $H_{3a}$: The map $\alpha\mapsto\varphi(\cdot,\alpha)$ is injective.
• $H_{3b}$: Each row and each column of $\boldsymbol{\alpha}^{\star}$ is unique.

[9] gives sufficient conditions for the generic identifiability of the categorical LBM, i.e. except on a manifold set of null Lebesgue measure in $\boldsymbol{\Theta}$ and this property is easily extended to the case of observations from a univariate exponential family. For binary SBM, [2] added the assumption $p\geq(\log n)/n$ on the parameter $p$ of the Bernoulli distribution to take into account sparsity.

Assumption $H_{1}$ ensures that the group proportions $\pi_{k}$ and $\rho_{\ell}$ are bounded away from [math] and $1$ so that no group disappears when $n$ and $d$ go to infinity. It also ensures that $\alpha$ is bounded away from the boundaries of $\mathcal{A}$ and that there exists a positive value $\kappa>0$, such that $[\alpha-\kappa,\alpha+\kappa]\subset\mathring{\mathcal{A}}$ for all parameters $\alpha$ of $C_{\alpha}$, which is essential to prove a uniform Bernstein inequality on the $X$.

Moreover, we define the quantity $\delta(\boldsymbol{\alpha})$ that captures the separation between row-groups or column-groups: low values of $\delta(\boldsymbol{\alpha})$ mean that two row-classes or two column-classes are very similar.

Definition 2.3** (class distinctness).**

For $\boldsymbol{\theta}=(\boldsymbol{\pi},\boldsymbol{\rho},\boldsymbol{\alpha})\in\boldsymbol{\Theta}$. We define:

[TABLE]

with $\operatorname{KL}(\alpha,\alpha^{\prime})=\mathbb{E}_{\alpha}[\log(\varphi(X,\alpha)/\varphi(X,\alpha^{\prime}))]=\psi^{\prime}(\alpha)(\alpha-\alpha^{\prime})+\psi(\alpha^{\prime})-\psi(\alpha)$ the Kullback divergence between $\varphi(.,\alpha)$ and $\varphi(.,\alpha^{\prime})$.

Remark 2.4*.*

Since $\boldsymbol{\alpha}^{\star}$ has distinct rows and distinct columns ($H_{3}$), $\delta(\boldsymbol{\alpha}^{\star})>0$.

Remark 2.5*.*

These assumptions are satisfied for many distributions, including but not limited to:

• •

Bernoulli, when the proportion $p$ is bounded away from [math] and $1$, or natural parameter $\alpha=\log(p/(1-p))$ bounded away from $\pm\infty$;

• •

Poisson, when the mean $\lambda$ is bounded away from [math] and $+\infty$, or natural parameter $\alpha=\log(\lambda)$ bounded away from $\pm\infty$;

• •

Gaussian with known variance when the mean $\mu$, which is also the natural parameter, is bounded away from $\pm\infty$.

In particular, the conditions stating that $\psi$ is twice differentiable and that $(\psi^{\prime})^{-1}$ exists are equivalent to assuming that $X_{\alpha}$ has positive and finite variance for all values of $\alpha$ in the parameter space.

2.3 Model Symmetry

The LBM is a generalized mixture model and as such is subject to label switching. Moreover, the study of the asymptotics will involve the complete likelihood where symmetry properties on the parameter must be taken into account. We first recall the definition of a permutation in LBM, then define equivalence relationships for assignments and parameter, and discuss model symmetry.

Definition 2.6** (permutation).**

Let $s$ be a permutation on $\{1,\dots,g\}$ and $t$ a permutation on $\{1,\dots,m\}$. If $\boldsymbol{A}$ is a matrix with $g$ columns, we define $\boldsymbol{A}^{s}$ as the matrix obtained by permuting the columns of $\boldsymbol{A}$ according to $s$, i.e. for any row $i$ and column $k$ of $\boldsymbol{A}$, ${A}^{s}_{ik}=A_{is(k)}$. If $\boldsymbol{B}$ is a matrix with $m$ columns and $\boldsymbol{C}$ is a matrix with $g$ rows and $m$ columns, $\boldsymbol{B}^{t}$ and $\boldsymbol{C}^{s,t}$ are defined similarly:

[TABLE]

Definition 2.7** (equivalence).**

We define the following equivalence relationships:

• •

Two assignments $(\mathbf{z},\mathbf{w})$ and $(\mathbf{z}^{\prime},\mathbf{w}^{\prime})$ are equivalent, denoted $\sim$, if they are equal up to label permutation, i.e. there exist two permutations $s$ and $t$ such that $\mathbf{z}^{\prime}=\mathbf{z}^{s}$ and $\mathbf{w}^{\prime}=\mathbf{w}^{t}$.

• •

Two parameters $\boldsymbol{\theta}$ and $\boldsymbol{\theta}^{\prime}$ are equivalent, denoted $\sim$, if they are equal up to label permutation, i.e. there exist two permutations $s$ and $t$ such that $(\boldsymbol{\pi}^{s},\boldsymbol{\rho}^{t},\boldsymbol{\alpha}^{s,t})=(\boldsymbol{\pi}^{\prime},\boldsymbol{\rho}^{\prime},\boldsymbol{\alpha}^{\prime})$. This is label-switching.

• •

$(\boldsymbol{\theta},\mathbf{z},\mathbf{w})$* and $(\boldsymbol{\theta}^{\prime},\mathbf{z}^{\prime},\mathbf{w}^{\prime})$ are equivalent, denoted $\sim$, if they are equal up to label permutation on $\boldsymbol{\alpha}$, i.e. there exist two permutations, $s$ and $t$ such that $(\boldsymbol{\alpha}^{s,t},\mathbf{z}^{s},\mathbf{w}^{t})=(\boldsymbol{\alpha}^{\prime},\mathbf{z}^{\prime},\mathbf{w}^{\prime})$.*

The last equivalence relationship is not concerned with $\boldsymbol{\pi}$ and $\boldsymbol{\rho}$. It is useful when dealing with the conditional likelihood $p(\mathbf{x}|\mathbf{z},\mathbf{w};\boldsymbol{\theta})$ which depends neither on $\boldsymbol{\pi}$ nor $\boldsymbol{\rho}$: in fact, if $(\boldsymbol{\theta},\mathbf{z},\mathbf{w})\sim(\boldsymbol{\theta}^{\prime},\mathbf{z}^{\prime},\mathbf{w}^{\prime})$, then for all $\mathbf{x}$, we have $p(\mathbf{x}|\mathbf{z},\mathbf{w};\boldsymbol{\theta})=p(\mathbf{x}|\mathbf{z}^{\prime},\mathbf{w}^{\prime};\boldsymbol{\theta}^{\prime})$. Note also that $\mathbf{z}\sim\mathbf{z}^{\star}$ (resp. $\mathbf{w}\sim\mathbf{w}^{\star}$) if and only if there exists a permutation of the rows of the confusion matrix ${I\!R}_{g}(\mathbf{z})$ (resp. ${I\!R}_{m}(\mathbf{w})$) leading to a diagonal matrix.

Definition 2.8** (symmetry).**

We say that the parameter $\boldsymbol{\theta}$ exhibits symmetry for the permutations $s,t$ if

[TABLE]

$\boldsymbol{\theta}$* exhibits symmetry if it exhibits symmetry for any non trivial pair of permutations $(s,t)$. Finally the set of pairs $(s,t)$ for which $\boldsymbol{\theta}$ exhibits symmetry is denoted $\operatorname{Sym}(\boldsymbol{\theta})$.*

Remark 2.9*.*

The set of parameters that exhibit symmetry is a manifold of null Lebesgue measure in $\boldsymbol{\Theta}$. This notion of symmetry is subtler than and different from label switching. To emphasize the difference between equivalence and symmetry, consider the following model: $\boldsymbol{\pi}=(1/2,1/2)$, $\boldsymbol{\rho}=(1/3,2/3)$ and \boldsymbol{\alpha}=\left(\begin{array}[]{cc}\alpha_{1}&\alpha_{2}\\ \alpha_{2}&\alpha_{1}\end{array}\right) with $\alpha_{1}\neq\alpha_{2}$. The only permutations of interest here are $s=t=[1\ 2]$. Choose any $\mathbf{z}$ and $\mathbf{w}$. Because of label switching, we know that $p(\mathbf{x},\mathbf{z}^{s},\mathbf{w}^{t};\boldsymbol{\theta}^{s,t})=p(\mathbf{x},\mathbf{z},\mathbf{w};\boldsymbol{\theta})$. $(\mathbf{z}^{s},\mathbf{w}^{t})$ and $(\mathbf{z},\mathbf{w})$ have the same likelihood but under different parameters $\boldsymbol{\theta}$ and $\boldsymbol{\theta}^{s,t}$. If however, $\boldsymbol{\rho}=(1/2,1/2)$, then $(s,t)\in\operatorname{Sym}(\boldsymbol{\theta})$ and $\boldsymbol{\theta}^{s,t}=\boldsymbol{\theta}$ so that $(\mathbf{z},\mathbf{w})$ and $(\mathbf{z}^{s},\mathbf{w}^{t})$ have the same likelihood under the same parameter $\boldsymbol{\theta}$. In particular, if $(\mathbf{z},\mathbf{w})$ is a maximum-likelihood assignment under $\boldsymbol{\theta}$, so is $(\mathbf{z}^{s},\mathbf{w}^{t})$. In other words, if $\boldsymbol{\theta}$ exhibits symmetry, the maximum-likelihood assignment is not unique under the true model and there are at least $\#\operatorname{Sym}(\boldsymbol{\theta})$ of them. This has important implications for the asymptotics of the observed likelihood ratio.

2.4 Distance and local assignments

We define the distance up to equivalence between two sets of assignments as follows:

Definition 2.10** (distance).**

The distance, up to equivalence, between configurations $\mathbf{z}$ and $\mathbf{z}^{\star}$ is defined as

[TABLE]

where, for all matrix $\mathbf{z}$, $\left\|\cdot\right\|_{0}$ is the Hamming norm

[TABLE]

A similar definition is set for the distance between $\mathbf{w}$ and $\mathbf{w}^{\star}$.

This allows us to define a neighborhood of radius $r$ in the assignment space, taking into account equivalent assignments classes.

Definition 2.11** (Set of local assignments).**

We denote $S(\mathbf{z}^{\star},\mathbf{w}^{\star},r)$ the set of configurations that have a representative (for $\sim$) within relative radius $r$ of $(\mathbf{z}^{\star},\mathbf{w}^{\star})$:

[TABLE]

3 Asymptotic properties in the complete data model

As stated in the introduction, we first study the asymptotic properties of the complete data model. Let $\widehat{\boldsymbol{\theta}}_{c}=\left(\widehat{\boldsymbol{\pi}},\widehat{\boldsymbol{\rho}},\widehat{\boldsymbol{\alpha}}\right)$ be the MLE of $\boldsymbol{\theta}$ in the complete data model, where the real assignments $\mathbf{z}=\mathbf{z}^{\star}$ and $\mathbf{w}=\mathbf{w}^{\star}$ are known. We can derive the following general estimates from Equation (2.1):

[TABLE]

Proposition 3.1**.**

The matrices $\Sigma_{\boldsymbol{\pi}^{\star}}=\operatorname{Diag}(\boldsymbol{\pi}^{\star})-\boldsymbol{\pi}^{\star}\left(\boldsymbol{\pi}^{\star}\right){{}^{T}}$, $\Sigma_{\boldsymbol{\rho}^{\star}}=\operatorname{Diag}(\boldsymbol{\rho}^{\star})-\boldsymbol{\rho}^{\star}\left(\boldsymbol{\rho}^{\star}\right){{}^{T}}$ are semi-definite positive, of rank $g-1$ and $m-1$, and $\widehat{\boldsymbol{\pi}}$ and $\widehat{\boldsymbol{\rho}}$ are asymptotically normal:

[TABLE]

*Similarly, let $V(\boldsymbol{\alpha}^{\star})$ be the matrix defined by $[V(\boldsymbol{\alpha}^{\star})]_{k\ell}=1/\psi^{\prime\prime}(\alpha^{\star}_{k\ell})$ and

$\Sigma_{\boldsymbol{\alpha}^{\star}}=\operatorname{Diag}^{-1}(\boldsymbol{\pi}^{\star})V(\boldsymbol{\alpha}^{\star})\operatorname{Diag}^{-1}(\boldsymbol{\rho}^{\star})$. Then:*

[TABLE]

and the components $\widehat{\alpha}_{k,\ell}$ are independent.

Proof: Since $\widehat{\boldsymbol{\pi}}\left(\mathbf{z}^{\star}\right)=\left(\widehat{\pi}_{1}\left(\mathbf{z}^{\star}\right),\dots,\widehat{\pi}_{g}\left(\mathbf{z}^{\star}\right)\right)$ (resp. $\widehat{\boldsymbol{\rho}}\left(\mathbf{w}^{\star}\right)$) is the sample mean of $n$ (resp. $d$) i.i.d. multinomial random variables with parameters $1$ and $\boldsymbol{\pi}^{\star}$ (resp. $\boldsymbol{\rho}^{\star}$), a simple application of the central limit theorem (CLT) gives:

[TABLE]

which proves Equation (3.2) where $\Sigma_{\boldsymbol{\pi}^{\star}}$ and $\Sigma_{\boldsymbol{\rho}^{\star}}$ are semi-definite positive of rank $g-1$ and $m-1$.

Similarly, $\psi^{\prime}\left(\widehat{\alpha}_{k\ell}\left(\mathbf{z}^{\star},\mathbf{w}^{\star}\right)\right)$ is the average of $z_{+k}^{\star}w_{+\ell}^{\star}=nd\widehat{\pi}_{k}\left(\mathbf{z}^{\star}\right)\widehat{\rho}_{\ell}\left(\mathbf{w}^{\star}\right)$ i.i.d. random variables with mean $\psi^{\prime}\left(\alpha^{\star}_{k\ell}\right)$ and variance $\psi^{\prime\prime}\left(\alpha^{\star}_{k\ell}\right)$. $nd\widehat{\pi}_{k}\left(\mathbf{z}^{\star}\right)\widehat{\rho}_{\ell}\left(\mathbf{w}^{\star}\right)$ is itself random but $\widehat{\pi}_{k}\left(\mathbf{z}^{\star}\right)\widehat{\rho}_{\ell}\left(\mathbf{w}^{\star}\right)\xrightarrow[n,d\to+\infty]{}\pi^{\star}_{k}\rho^{\star}_{\ell}$ almost surely. Therefore, by Slutsky’s lemma and the CLT for random sums of random variables [12], we have:

[TABLE]

The differentiability of $(\psi^{\prime})^{-1}$ and the delta method then gives:

[TABLE]

and the independence results from the independence of $\widehat{\alpha}_{k\ell}\left(\mathbf{z}^{\star},\mathbf{w}^{\star}\right)$ and $\widehat{\alpha}_{k^{\prime}\ell^{\prime}}\left(\mathbf{z}^{\star},\mathbf{w}^{\star}\right)$ as soon as $k\neq k^{\prime}$ or $\ell\neq\ell^{\prime}$, as they involve different sets of independent variables.

$\square$

Moreover, the complete model is locally asymptotically normal (LAN), as stated in the following proposition. Note that the unusual condition for $s$ and $t$ arises from the constraints $\boldsymbol{\pi}^{{}^{T}}\mathbf{1}_{g}=\boldsymbol{\rho}^{{}^{T}}\mathbf{1}_{m}=1$, where $\mathbf{1}_{g}$ is the vector of size $g$ filled with $1$, which must be satisfied even after perturbing $\boldsymbol{\pi}$ (resp. $\boldsymbol{\rho}$ ) with $s$ (resp. $t$).

Proposition 3.2** (Local asymptotic normality).**

Let $\mathcal{L}_{c}^{\star}$ the map defined by $\boldsymbol{\theta}=\left(\boldsymbol{\pi},\boldsymbol{\rho},\boldsymbol{\alpha}\right)\mapsto\log p\left(\mathbf{x},\mathbf{z}^{\star},\mathbf{w}^{\star};\boldsymbol{\theta}\right)$ and note $I_{\boldsymbol{\pi}^{\star}}=\operatorname{Diag}^{-1}(\boldsymbol{\pi}^{\star})$, $I_{\boldsymbol{\rho}^{\star}}=\operatorname{Diag}^{-1}(\boldsymbol{\rho}^{\star})$ and $I_{\boldsymbol{\alpha}^{\star}}$ the component-wise inverse of $\Sigma_{\boldsymbol{\alpha}^{\star}}$. For any $s$, $t$ and $u$ in a compact set , such that $t^{{}^{T}}\mathbf{1}_{g}=0$ and $s^{{}^{T}}\mathbf{1}_{m}=0$, we have:

[TABLE]

*where $\odot$ denotes the Hadamard product of two matrices (element-wise product), $\mathbf{Y}_{\boldsymbol{\pi}^{\star}}$, $\mathbf{Y}_{\boldsymbol{\rho}^{\star}}$ are asymptotically centered Gaussian vectors of sizes $g$ and $m$ with respective variance matrices $I(\boldsymbol{\pi}^{\star})$ and $I(\boldsymbol{\rho}^{\star})$ and $\mathbf{Y}_{\boldsymbol{\alpha}^{\star}}$ is a random matrix of size $g\times m$ with independent Gaussian components $Y_{\boldsymbol{\alpha}^{\star},kl}\sim{\cal N}(0,I_{\boldsymbol{\alpha}^{\star},kl})$. *

Proof.

By Taylor expansion, and with the condition $s^{{}^{T}}\mathbf{1}_{g}=t^{{}^{T}}\mathbf{1}_{m}=0$

[TABLE]

where $\nabla{\mathcal{L}_{c}^{\star}}_{\boldsymbol{\pi}}\left(\boldsymbol{\theta}^{\star}\right)$, $\nabla{\mathcal{L}_{c}^{\star}}_{\boldsymbol{\rho}}\left(\boldsymbol{\theta}^{\star}\right)$ and $\nabla{\mathcal{L}_{c}^{\star}}_{\boldsymbol{\alpha}}\left(\boldsymbol{\theta}^{\star}\right)$ denote the respective components of the gradient of $\mathcal{L}_{c}^{\star}$ evaluated at $\boldsymbol{\theta}^{\star}$ and $\mathbf{H}_{\boldsymbol{\pi}}$, $\mathbf{H}_{\boldsymbol{\rho}}$ and $\mathbf{H}_{\boldsymbol{\alpha}}$ denotes the conditional hessian of $\mathcal{L}_{c}^{\star}$ evaluated at $\boldsymbol{\theta}^{\star}$. By inspection, $\mathbf{H}_{\boldsymbol{\pi}}/n$, $\mathbf{H}_{\boldsymbol{\rho}}/d$ and $\mathbf{H}_{\boldsymbol{\alpha}}/nd$ converge in probability to constant matrices and the random vectors $\nabla{\mathcal{L}_{c}^{\star}}_{\boldsymbol{\pi}}\left(\boldsymbol{\theta}^{\star}\right)/\sqrt{n}$, $\nabla{\mathcal{L}_{c}^{\star}}_{\boldsymbol{\rho}}\left(\boldsymbol{\theta}^{\star}\right)/\sqrt{d}$ and $\nabla{\mathcal{L}_{c}^{\star}}_{\boldsymbol{\alpha}}\left(\boldsymbol{\theta}^{\star}\right)/\sqrt{nd}$ converge in distribution to Gaussian vectors by the central limit theorem.

$\square$

4 Profile Likelihood

Our main result compares the observed likelihood ratio $p(\mathbf{x};\boldsymbol{\theta})/p(\mathbf{x};\boldsymbol{\theta}^{\star})$ with the complete likelihood $p(\mathbf{x},\mathbf{z}^{\star},\mathbf{w}^{\star};\boldsymbol{\theta})/p(\mathbf{x},\mathbf{z}^{\star},\mathbf{w}^{\star};\boldsymbol{\theta}^{\star})$. To study the behavior of these likelihoods, we shall work conditionally to the true configurations $(\mathbf{z}^{\star},\mathbf{w}^{\star})$ that have enough observations in each row or column group. We therefore define in section 4.1 so called regular configurations and prove that they occur with high probability. We then introduce in section 4.2 conditional and profile log-likelihood ratios and state some of their properties.

4.1 Regular assignments

Definition 4.1** ($c$-regular assignments).**

Let $\mathbf{z}\in\mathcal{Z}$ and $\mathbf{w}\in\mathcal{W}$. For any $c>0$, we say that $\mathbf{z}$ and $\mathbf{w}$ are c-regular if

[TABLE]

In regular configurations, each row-group for example has $\Omega(n)$ members, where $u_{n}=\Omega(n)$ if there exists two constant $a,b>0$ such that for $n$ enough large $an\leq u_{n}\leq bn$. $c/2$-regular assignments, with $c$ defined in Assumption $H_{1}$, have high $\mathbb{P}_{\boldsymbol{\theta}^{\star}}$-probability in the space of all assignments, uniformly over all $\boldsymbol{\theta}^{\star}\in\boldsymbol{\Theta}$, as stated in Proposition 4.2.

Proposition 4.2**.**

Define $\mathcal{Z}_{1}$ and $\mathcal{W}_{1}$ as the subsets of $\mathcal{Z}$ and $\mathcal{W}$ made of $c/2$-regular assignments, with $c$ defined in assumption $H_{1}$. Denote $\Omega_{1}$ the event $\{(\mathbf{z}^{\star},\mathbf{w}^{\star})\in\mathcal{Z}_{1}\times\mathcal{W}_{1}\}$, then:

[TABLE]

Each $z_{+k}$ is a sum of $n$ i.i.d Bernoulli random variables with parameter $\pi_{k}\geq\pi_{\min}\geq c$. The proof is straightforward and stems from a simple Hoeffding bound

[TABLE]

and a union bound over $g$ values of $k$, with similar approach for $w_{+\ell}$.

4.2 Conditional and profile log-likelihoods

Introducing the conditional log-likelihood ratio

[TABLE]

the complete likelihood can be written as follows

[TABLE]

The study of $F_{nd}$ will be of crucial importance, as well as its maximum over $\boldsymbol{\Theta}$. After some definitions, we examine some useful properties.

Definition 4.3**.**

The conditional expectation $G$ of $F_{nd}$ is defined as:

[TABLE]

Moreover, the profile log-likelihood ratio $\Lambda$ and its expectation $\tilde{\Lambda}$ are defined as:

[TABLE]

Remark 4.4*.*

As $F_{nd}$ and $G$ only depend on $\boldsymbol{\theta}$ through $\boldsymbol{\alpha}$, we will sometimes replace $\boldsymbol{\theta}$ with $\boldsymbol{\alpha}$ in the expressions of $F_{nd}$ and $G$. Replacing $F_{n,d}$ and $G$ by their profiled version $\Lambda$ and $\tilde{\Lambda}$ allows us to get rid of the continuous argument $\boldsymbol{\alpha}$ of $F_{nd}$ and to rely instead only on discrete contrasts $\Lambda$ and $\tilde{\Lambda}$.

Now, Proposition 4.5 characterizes which values of $\boldsymbol{\alpha}$ maximize $F_{nd}$ and $G$ to reach $\Lambda$ and $\tilde{\Lambda}$. Propositions 4.6 and 4.7 in turn describes properties of $G$ and $\tilde{\Lambda}$ relative to $(\mathbf{z},\mathbf{w})$.

Proposition 4.5** (maximum of $G$ and $\tilde{\Lambda}$ in $\theta$).**

Let $\widehat{\theta}_{c}=(\widehat{x}_{k\ell}(\mathbf{z},\mathbf{w}),\widehat{\pi}_{k}(\mathbf{z}),\widehat{\rho}_{\ell}(\mathbf{w}))$ be the maximum likelihood estimator of the complete model, as defined in Equation 3.1. Conditionally on $\mathbf{z}^{\star},\mathbf{w}^{\star}$, define the following quantities:

[TABLE]

with $\bar{x}_{k\ell}(\mathbf{z},\mathbf{w})=0$ for $\mathbf{z}$ and $\mathbf{w}$ such that $\widehat{\pi}_{k}(\mathbf{z})=0$ or $\widehat{\rho}_{\ell}(\mathbf{w})=0$.

Then $F_{nd}(\boldsymbol{\theta},\mathbf{z},\mathbf{w})$ (resp. $G(\boldsymbol{\theta},\mathbf{z},\mathbf{w})$) is maximum in $\boldsymbol{\alpha}$ for $\boldsymbol{\alpha}=\widehat{\boldsymbol{\alpha}}(\mathbf{z},\mathbf{w})$ (resp. $\bar{\boldsymbol{\alpha}}(\mathbf{z},\mathbf{w})$) defined by:

[TABLE]

Hence,

[TABLE]

Note that although $\bar{x}_{k\ell}=\mathbb{E}_{\boldsymbol{\theta}^{\star}}\left[\left.\widehat{x}_{k\ell}\right|\mathbf{z}^{\star},\mathbf{w}^{\star}\right]$, in general $\bar{\alpha}_{k\ell}\neq\mathbb{E}_{\boldsymbol{\theta}^{\star}}\left[\left.\widehat{\alpha}_{k\ell}\right|\mathbf{z}^{\star},\mathbf{w}^{\star}\right]$ by non linearity of $(\psi^{\prime})^{-1}$. Nevertheless, since $(\psi^{\prime})^{-1}$ is Lipschitz over compact subsets of $\psi^{\prime}(\mathring{\mathcal{A}})$, with high probability, $|\bar{\alpha}_{k\ell}-\widehat{\alpha}_{k\ell}|$ and $|\widehat{x}_{k\ell}-\bar{x}_{k\ell}|$ are of the same order of magnitude.

Proposition 4.6** (maximum of $G$ and $\tilde{\Lambda}$ in $(\theta,\mathbf{z},\mathbf{w})$).**

Let $\operatorname{KL}(\alpha,\alpha^{\prime})=\psi^{\prime}(\alpha)(\alpha-\alpha^{\prime})+\psi(\alpha^{\prime})-\psi(\alpha)$ be the Kullback divergence between $\varphi(.,\alpha)$ and $\varphi(.,\alpha^{\prime})$ then:

[TABLE]

Conditionally on the set $\Omega_{1}$ of regular assignments and for $n,d>2/c$,

• (i)

$G$* is maximized at $(\boldsymbol{\alpha}^{\star},\mathbf{z}^{\star},\mathbf{w}^{\star})$ and its equivalence class.*

• (ii)

$\tilde{\Lambda}$* is maximized at $(\mathbf{z}^{\star},\mathbf{w}^{\star})$ and its equivalence class and $\tilde{\Lambda}(\mathbf{z}^{\star},\mathbf{w}^{\star})=0$.*

Moreover, the maximum of $\tilde{\Lambda}$ in $(\mathbf{z}^{\star},\mathbf{w}^{\star})$ is well separated, in the sense that there exists a positive gap between $\tilde{\Lambda}(\mathbf{z}^{\star},\mathbf{w}^{\star})$ and any other $\tilde{\Lambda}(\mathbf{z},\mathbf{w})$ for $(\mathbf{z},\mathbf{w})$ in a close neighborhood of $(\mathbf{z}^{\star},\mathbf{w}^{\star})$, as stated in the following proposition:

Proposition 4.7** (Separability for $\tilde{\Lambda}$).**

Conditionally upon $\Omega_{1}$, there exists a positive constant $C$ such that for all $(\mathbf{z},\mathbf{w})\in S(\mathbf{z}^{\star},\mathbf{w}^{\star},C)$:

[TABLE]

Moreover, there exists a positive constant $B(C)$ such that for all $(\mathbf{z},\mathbf{w})\notin S(\mathbf{z}^{\star},\mathbf{w}^{\star},C)$

[TABLE]

The proofs of these propositions are reported in Appendix A. Proof of Proposition 4.5 follows from a straightforward calculation, proof of Proposition 4.6 uses the technical Lemma D.1 to characterize the maximum of $G$ and proof of Proposition 4.7 uses regularity properties of the gradient of $\tilde{\Lambda}$ to control its behavior near its maximum.

5 Main Result

Our main result matches the asymptotics of complete and observed likelihoods and is the key to prove the consistency of maximum likelihood and variational estimators. It is set under the assumptions described in section 2.2 and the following asymptotics for the number of rows $n$ and columns $d$:

[TABLE]

Theorem 5.1** (complete-observed).**

Let $\mathbf{x}$ be a matrix of $n\times d$ observations of a LBM with true parameter $\boldsymbol{\theta}^{\star}=(\boldsymbol{\pi}^{\star},\boldsymbol{\rho}^{\star},\boldsymbol{\alpha}^{\star})$ where the number of row-groups $g$ and column-groups $m$ are known, which conditional distribution belongs to a regular univariate exponential family. The true random and unobserved assignations for rows and columns are denoted $\mathbf{z}^{\star}$ and $\mathbf{w}^{\star}$ respectively. Define $\#\operatorname{Sym}(\boldsymbol{\theta})$ as the number of pairs of permutations $(s,t)$ for which $\boldsymbol{\theta}$ exhibits symmetry.

If assumptions $H_{1}$ to $H_{4}$ are fulfilled, then, the observed likelihood ratio behaves like the complete likelihood ratio, up to a bounded multiplicative factor:

[TABLE]

where both $o_{P}$ are uniform over all $\boldsymbol{\theta}\in\boldsymbol{\Theta}$.

The maximum over all $\boldsymbol{\theta}^{\prime}$ that are equivalent to $\boldsymbol{\theta}$ stems from the fact that because of label-switching, $\boldsymbol{\theta}$ is only identifiable up to its $\sim$-equivalence class from the observed likelihood, whereas it is completely identifiable from the complete likelihood as in this latter case, the labels are known. The terms $\#\operatorname{Sym}$ are needed to take into account cases where $\boldsymbol{\theta}$ exhibits symmetry. These were omitted by [2] for SBM, although they are also needed in this case, see remark 5.3. When no $\boldsymbol{\theta}\in\boldsymbol{\Theta}$ exhibits symmetry, the following corollary is immediately deduced :

Corollary 5.2**.**

If $\boldsymbol{\Theta}$ contains only parameters that do not exhibit symmetry:

[TABLE]

where the $o_{P}$ is uniform over all $\boldsymbol{\theta}\in\boldsymbol{\Theta}$.

General sketch of the proof.

The proof relies on the following decomposition of the observed likelihood:

[TABLE]

where the second term shall be proved to be asymptotically negligible. Its control stems from the study of the conditional log-likelihood $F_{nd}$, see Equation 4.2. In fact, the contribution of configurations that are not equivalent to $(\mathbf{z}^{\star},\mathbf{w}^{\star})$ leads itself to the study of a global control, and a sharper local control of $F_{nd}$. Hence, the proof relies on the examination of the asymptotic behavior of $F_{nd}$ on three types of configurations that partition $\mathcal{Z}\times\mathcal{W}$:

global control for assignations $(\mathbf{z},\mathbf{w})$ sufficiently far from $(\mathbf{z}^{\star},\mathbf{w}^{\star})$, i.e. such that $\tilde{\Lambda}(\mathbf{z},\mathbf{w})$ is of order $\Omega(-nd)$. Proposition 5.5 gives a large deviation result for $F_{nd}-\tilde{\Lambda}(\mathbf{z},\mathbf{w})$ to prove that $F_{nd}$ is also of order $-\Omega_{P}(nd)$. A key point will be the use of Proposition C.4, establishing a specific concentration inequality for sub-exponential variables. In turn, those assignments contribute as a $o_{P}(p(\mathbf{x},\mathbf{z}^{\star},\mathbf{w}^{\star};\boldsymbol{\theta}^{\star}))$ to the sum (Proposition 5.6). 2. 2.

local control: a small deviation result (Proposition 5.7) is needed to show that the combined contribution of assignments close to but not equivalent to $(\mathbf{z}^{\star},\mathbf{w}^{\star})$ is also a $o_{P}(p(\mathbf{x},\mathbf{z}^{\star},\mathbf{w}^{\star};\boldsymbol{\theta}^{\star}))$ (Proposition 5.8). 3. 3.

equivalent assignments: Proposition 5.9 examines which of the remaining assignments, all equivalent to $(\mathbf{z}^{\star},\mathbf{w}^{\star})$, contribute to the sum.

Once these propositions proved, the proof is straightforward, as can be seen below. They are in turn carefully presented and discussed in dedicated subsections as they represents the core arguments and their proofs are themselves postponed to Appendix B for more readability.

Proof.

We work conditionally to $\Omega_{1}$, defined in Proposition 4.2, i.e., the high probability event that $(\mathbf{z}^{\star},\mathbf{w}^{\star})$ is a $c/2$-regular assignment. We choose $(\mathbf{z}^{\star},\mathbf{w}^{\star})\in\mathcal{Z}_{1}\times\mathcal{W}_{1}$ and a sequence $t_{nd}$ decreasing to [math] but satisfying $t_{nd}^{2}\gg\frac{n+d}{nd}$. This is possible when $n\rightarrow\infty$ and $d\rightarrow\infty$, and for example with Assumption $(H_{4})$. We write:

[TABLE]

According to Proposition 5.6, conditionally to $\Omega_{1}$ and for $n,d$ large enough that $2\sqrt{2nd}t_{nd}\geq gm$, the contribution of far away assignments is

[TABLE]

Using the separability of $\tilde{\Lambda}$ and Assumption $(H_{4})$, Proposition 5.8 ensures the existence of $C>0$ such that:

[TABLE]

Since $t_{nd}$ decreases to [math], Proposition 5.8 can be applied for the local configurations belonging to $S(t_{nd})$, for $n,d$ large enough. Therefore the observed likelihood ratio reduces to:

[TABLE]

Proposition 5.9 deals with equivalence and symmetry and allows us to conclude

[TABLE]

$\square$

Remark 5.3*.*

As already pointed out, if $\boldsymbol{\theta}$ exhibits symmetry, the maximum likelihood assignment is not unique under $\boldsymbol{\theta}$, and $\#\operatorname{Sym}(\boldsymbol{\theta})$ terms contribute with the same weight. This was not taken into account by [2], and it is interesting to see why it should be also present for SBM. Recall that SBM has only one set of labels $\mathbf{z}$. The proof relies on the the decomposition

[TABLE]

where the second term of the sum is neglectible compared to the first term. Now, $\mathbf{z}^{\prime}\sim\mathbf{z}^{\star}$ means that there exists a permutation $t:[g]\rightarrow[g]$ such that $\mathbf{z}^{\prime}=\mathbf{z}^{t}$ and $p(\mathbf{x},\mathbf{z}^{t};\boldsymbol{\theta})=p(\mathbf{x},\mathbf{z};\boldsymbol{\theta}^{t})$. The first term is written on Page 1941, Equation (25) in [2] as

[TABLE]

However, the first equality is not always correct. Actually, we have

[TABLE]

Take a special case of symmetry where $\pi=(1/g,\ldots,1/g)$ and $\alpha=(p-q)I_{g}+q1_{g}1_{g}^{T}$. Then we have $\boldsymbol{\theta}^{t}=\boldsymbol{\theta}$ for all $t$. Thus,

[TABLE]

Even for the SBM, we thus have generally:

[TABLE]

5.1 Global Control

A large deviation inequality for configurations $(\mathbf{z},\mathbf{w})$ far from $(\mathbf{z}^{\star},\mathbf{w}^{\star})$ is build and used to prove that far away configurations make a small contribution to $p(\mathbf{x};\boldsymbol{\theta})$. Since we restricted $\alpha$ in a bounded subset of $\mathring{\mathcal{A}}$, there exists two positive values $M_{\alpha}$ and $\kappa$ such that $C_{\alpha}+(-\kappa,\kappa)\subset[-M_{\alpha},M_{\alpha}]\subset\mathring{\mathcal{A}}$. Moreover, the variance of $X_{\alpha}$ is bounded away from [math] and $+\infty$:

[TABLE]

Proposition 5.4**.**

With the previous notations, if $\alpha\in C_{\alpha}$ and $X_{\alpha}\sim\varphi(.,\alpha)$, then $X_{\alpha}$ is sub-exponential with parameters $(\bar{\sigma}^{2},\kappa^{-1})$.

The latter proposition is a direct consequence of the definition of sub-exponential variables, see Appendix C.

Proposition 5.5** (large deviations of $F_{nd}$).**

Let $\operatorname{Diam}(\boldsymbol{\Theta})=\sup_{\boldsymbol{\theta},\boldsymbol{\theta}^{\prime}}\|\boldsymbol{\theta}-\boldsymbol{\theta}^{\prime}\|_{\infty}$. For all $\varepsilon_{n,d}\leq\kappa\bar{\sigma}$ and $n,d$

[TABLE]

In particular, if $n$ and $d$ are large enough that $2\sqrt{2nd}\varepsilon_{nd}\geq gm$, the previous inequality ensures that with high probability, $F_{nd}(\boldsymbol{\theta},\mathbf{z},\mathbf{w})-\tilde{\Lambda}(\mathbf{z},\mathbf{w})$ is no greater than $\bar{\sigma}nd\operatorname{Diam}(\boldsymbol{\Theta})4\sqrt{2}\varepsilon_{nd}$.

The concentration inequality used in [2] to prove an analog result for SBM is not sufficient here, as it can be used only for upper-bounded observations, which is obviously not the case for all exponential families. We instead develop a Bernstein-type inequality for sub-exponential variables (Proposition C.4) to upper bound $F_{nd}(\boldsymbol{\theta},\mathbf{z},\mathbf{w})-\tilde{\Lambda}(\mathbf{z},\mathbf{w})$. Proposition 5.5 relies heavily on this Bernstein inequality. A straightforward consequence of this deviation bound is that the combined contribution of assignments far away from $(\mathbf{z}^{\star},\mathbf{w}^{\star})$ to the sum is negligible, assuming that the numbers $n$ of rows and $d$ of columns grow at commensurate rates, as stated in the following proposition:

Proposition 5.6** (contribution of far away assignments).**

Assume $n\to\infty$ and $d\to\infty$, and choose $t_{nd}$ decreasing to [math] such that $t_{nd}^{2}\gg\frac{n+d}{nd}$. Then conditionally on $\Omega_{1}$ and for $n,d$ large enough that $2\sqrt{2nd}t_{nd}\geq gm$, we have:

[TABLE]

where the $o_{P}$ is uniform in probability over all $\boldsymbol{\theta}\in\boldsymbol{\Theta}$.

5.2 Local Control

Proposition 5.5 gives deviations of order $\mathcal{O}_{P}(\sqrt{nd})$, which are only useful for $(\mathbf{z},\mathbf{w})$ such that $G$ and $\tilde{\Lambda}$ are large compared to $\sqrt{nd}$. For $(\mathbf{z},\mathbf{w})$ close to $(\mathbf{z}^{\star},\mathbf{w}^{\star})$, we need tighter concentration inequalities, of order $o_{P}(n+d)$, as follows:

Proposition 5.7** (small deviations $F_{nd}$).**

Conditionally upon $\Omega_{1}$, for $n$ and $d$ satisfying $(H_{4})$, and for $(\mathbf{z},\mathbf{w})\in S(\mathbf{z}^{\star},\mathbf{w}^{\star},c/4)$ where $c>0$ is defined in $(H_{1})$, we have:

[TABLE]

The next proposition uses Propositions 4.6 and 5.7 to show that the combined contribution to the observed likelihood of assignments close to $(\mathbf{z}^{\star},\mathbf{w}^{\star})$ is also a $o_{P}$ of $p(\mathbf{z}^{\star},\mathbf{w}^{\star},\mathbf{x};\boldsymbol{\theta}^{\star})$:

Proposition 5.8** (contribution of local assignments).**

With the previous notations and for $n$ and $d$ satisfying Assumption $(H_{4})$, for any $\tilde{c}\leq\min(C,c/4)$ we have:

[TABLE]

5.3 Equivalent assignments

It remains to study the contribution of equivalent assignments.

Proposition 5.9** (contribution of equivalent assignments).**

For all $\boldsymbol{\theta}\in\boldsymbol{\Theta}$, we have

[TABLE]

where the $o_{P}$ is uniform in $\boldsymbol{\theta}$.

The maximum over $\boldsymbol{\theta}^{\prime}\sim\boldsymbol{\theta}$ accounts for equivalent configurations whereas $\#\operatorname{Sym}(\boldsymbol{\theta})$ is needed when $\boldsymbol{\theta}$ exhibits symmetry, as noticed in Remark 5.3.

5.4 Evolution of parameters

This is convenient in the binary setting where increasing sparsity [math]s can be done directly by scaling the $p_{ij}$s with a common factor. In the valued setting however, this approach fails to model actual observations and the equivalent is to consider the product of a Bernoulli variable with the actual observation value. We choose not to explore this direction and consider instead sparsity settings where the numbers $n$ of rows and $d$ columns grow at very different rates.

To simplify the reading of the proofs, the parameters $c$ (used in the assumption $H_{1}$), $\delta(\boldsymbol{\alpha}^{\star})$, $g$ and $m$ are supposed to be fixed. However, the results of the theorem 5.1 remain true even if they evolve with the following derivated assumptions:

1. $H_{1}^{n,d}$

: There exist three sequences of positive constants $c_{n,d}$, $\kappa_{n,d}$ and $M_{\alpha,n,d}$, and a sequence of compacts $C_{\alpha,n,d}$ such that

[TABLE] 2. $H_{2}^{n,d}$

: The true parameters $\boldsymbol{\theta}^{\star}_{n,d}=(\boldsymbol{\pi}^{\star}_{n,d},\boldsymbol{\rho}^{\star}_{n,d},\boldsymbol{\alpha}^{\star}_{n,d})$ lies in the relative interior of $\boldsymbol{\Theta}_{n,d}$. 3. $H_{3}^{n,d}$

: $\boldsymbol{\theta}^{\star}_{n,d}$ are identifiable. 4. $H_{4}^{n,d}$

: If we denote $\bar{\sigma}_{{n,d}}^{2}=\sup_{\alpha\in[-M_{\alpha,n,d},M_{\alpha,n,d}]}\mathbb{V}(X_{\alpha})$, the parameters satisfy the following conditions:

[TABLE]

Remark 5.10*.*

There is no condition on $g_{n,d}$ or $m_{n,d}$ because the number of cluster is bounded by $1/c_{n,d}$.

Remark 5.11*.*

In their article, [2] have some conditions on the sparsity

Corollary 5.12**.**

Consider that assumptions $H_{1}^{n,d}$ to $H_{4}^{n,d}$ hold for the Latent Block Model of known order with $n\times d$ observations coming from an univariate exponential family and define $\#\operatorname{Sym}(\boldsymbol{\theta})$ as the set of pairs of permutation $(s,t)$ for which $\boldsymbol{\theta}=(\boldsymbol{\pi},\boldsymbol{\rho},\boldsymbol{\alpha})$ exhibits symmetry. Then, for $n$ and $d$ tending to infinity with asymptotic rates $\log(d)/n\rightarrow 0$ and $\log(n)/d\rightarrow 0$, the observed likelihood ratio behaves like the complete likelihood ratio, up to a bounded multiplicative factor:

[TABLE]

where the $o_{P}$ is uniform over all $\boldsymbol{\theta}\in\boldsymbol{\Theta}$. Reste vrai sur $\boldsymbol{\Theta}_{n,d}$ ?

6 Asymptotics for the Maximum Likelihood (MLE) and Variational (VE) Estimators

This section is devoted to the asymptotics of the MLE and VE in the incomplete data model as a consequence of the main result 5.1.

6.1 ML estimator

Theorem 6.1** (Asymptotic behavior of $\widehat{\boldsymbol{\theta}}_{MLE}$).**

Denote $\widehat{\boldsymbol{\theta}}_{MLE}$ the maximum likelihood estimator and use the notations of Proposition 3.1. There exist permutations $s$ of $\{1,\dots,g\}$ and $t$ of $\{1,\dots,m\}$ such that

[TABLE]

The proof relies on a Taylor expansion of the complete likelihood near its optimum, like in Proposition 3.2, and on our main theorem.

Proof.

Note first that unless $\boldsymbol{\Theta}$ is constrained and with high probability, $\widehat{\boldsymbol{\theta}}_{MLE}$ and $\widehat{\boldsymbol{\theta}}_{c}(\mathbf{z}^{\star},\mathbf{w}^{\star})$ exhibit no symmetries. Indeed, equalities like $\widehat{x}_{k\ell}=\widehat{x}_{k^{\prime},\ell^{\prime}}$ have vanishingly small probabilities of being simultaneously true when $X_{ij}$ is discrete and null when $X_{ij}$ is continuous.

Note also that $\mathcal{L}_{c}^{\star}$ has a unique maximum at $\widehat{\boldsymbol{\theta}}_{c}(\mathbf{z}^{\star},\mathbf{w}^{\star})$. Furthermore, the curvature of $\mathcal{L}_{c}^{\star}$ at $\widehat{\boldsymbol{\theta}}_{c}(\mathbf{z}^{\star},\mathbf{w}^{\star})$ with respect to $\boldsymbol{\pi}$ (resp. $\boldsymbol{\rho}$, $\boldsymbol{\alpha}$) converge in probability to $I_{\boldsymbol{\pi}^{\star}}$ (resp. $I_{\boldsymbol{\rho}^{\star}}$, $I_{\boldsymbol{\alpha}^{\star}}$) defined in Proposition 3.2 by consistency of $\widehat{\boldsymbol{\theta}}_{c}$. Therefore any estimator $\widehat{\boldsymbol{\theta}}$ bounded away from $\widehat{\boldsymbol{\theta}}_{c}(\mathbf{z}^{\star},\mathbf{w}^{\star})$ satisfies $\mathcal{L}_{c}^{\star}(\widehat{\boldsymbol{\theta}}_{c}(\mathbf{z}^{\star},\mathbf{w}^{\star}))-\mathcal{L}_{c}^{\star}(\widehat{\boldsymbol{\theta}})>\Omega_{P}(1)$. If $\|\widehat{\boldsymbol{\theta}}_{c}(\mathbf{z}^{\star},\mathbf{w}^{\star})-\widehat{\boldsymbol{\theta}}\|=o_{P}(1)$, a Taylor expansion at $\widehat{\boldsymbol{\theta}}_{c}(\mathbf{z}^{\star},\mathbf{w}^{\star})$ gives

[TABLE]

where the linear term in the expansion vanishes as $\widehat{\boldsymbol{\theta}}_{c}(\mathbf{z}^{\star},\mathbf{w}^{\star})$ is the argmax of $\mathcal{L}_{c}^{\star}$ and the Hessian of $\mathcal{L}_{c}^{\star}$ at $\widehat{\boldsymbol{\theta}}_{c}(\mathbf{z}^{\star},\mathbf{w}^{\star})$ were replaced by their limit in probability.

We may now prove the corollary by contradiction. Assume that $\min_{s}(\widehat{\boldsymbol{\pi}}_{MLE}^{s}-\widehat{\boldsymbol{\pi}}\left(\mathbf{w}^{\star}\right))\neq o_{P}\left(n^{-1/2}\right)$, $\min_{t}(\widehat{\boldsymbol{\rho}}_{MLE}^{t}-\widehat{\boldsymbol{\rho}}\left(\mathbf{w}^{\star}\right))\neq o_{P}\left(d^{-1/2}\right)$ or $\min_{s,t}(\widehat{\boldsymbol{\alpha}}_{MLE}^{s,t}-\widehat{\boldsymbol{\alpha}}\left(\mathbf{z}^{\star},\mathbf{w}^{\star}\right))\neq o_{P}\left(nd^{-1/2}\right)$ where $s$ and $t$ are permutations of $\{1,\dots,g\}$ and $\{1,\dots,m\}$. Plugging $\widehat{\boldsymbol{\theta}}_{MLE}^{s,t}$ in the previous expansion shows that:

[TABLE]

But, since $\widehat{\boldsymbol{\theta}}_{c}\left(\mathbf{z}^{\star},\mathbf{w}^{\star}\right)$ and $\widehat{\boldsymbol{\theta}}_{MLE}$ maximize respectively $\frac{p(\mathbf{x},\mathbf{z}^{\star},\mathbf{w}^{\star};\boldsymbol{\theta})}{p(\mathbf{x},\mathbf{z}^{\star},\mathbf{w}^{\star};\boldsymbol{\theta}^{\star})}$ and $\frac{p(\mathbf{x};\boldsymbol{\theta})}{p\left(\mathbf{x};\boldsymbol{\theta}^{\star}\right)}$ and have no symmetries, it follows by Theorem 5.1 that

[TABLE]

which contradicts Equation (6.1) and concludes the proof.

$\square$

6.2 Variational estimator

Due to the complex dependence structure of the observations, the maximum likelihood estimator of the LBM is not numerically tractable, even with the EM-algorithm. In practice, a variational approximation can be used, see for example [5]: for any joint distribution $\mathbb{Q}\in\mathcal{Q}$ on $\mathcal{Z}\times\mathcal{W}$ a lower bound of $\mathcal{L}(\boldsymbol{\theta})$ is given by

[TABLE]

where $\mathcal{H}\left(\mathbb{Q}\right)=-\mathbb{E}_{\mathbb{Q}}[\log(\mathbb{Q})]$. Choose $\mathcal{Q}$ to be the set of factorized distributions, such that for all $\left(\mathbf{z},\mathbf{w}\right)$

[TABLE]

allows to obtain tractable expressions of $J\left(\mathbb{Q},\boldsymbol{\theta}\right)$ as a lower bound of the log-likelihood. The variational estimate $\widehat{\boldsymbol{\theta}}_{var}$ of $\boldsymbol{\theta}$ is defined as

[TABLE]

The following corollary states that $\widehat{\boldsymbol{\theta}}_{var}$ has the same asymptotics as $\widehat{\boldsymbol{\theta}}_{MLE}$ and $\widehat{\boldsymbol{\theta}}_{c}$.

Theorem 6.2** (Variational estimate).**

Under the assumptions of Theorem 5.1 there exist permutations $s$ of $\{1,\dots,g\}$ and $t$ of $\{1,\dots,m\}$ such that

[TABLE]

oreover,

[TABLE]

where the $o_{P}$ is uniform over all $\boldsymbol{\theta}\in\boldsymbol{\Theta}\backslash\left\{\boldsymbol{\theta}\in\boldsymbol{\Theta}|\#\operatorname{Sym}(\boldsymbol{\theta})=1\right\}$.

Proof.

Remark first that for every $\boldsymbol{\theta}$ and for every $\left(\mathbf{z},\mathbf{w}\right)$,

[TABLE]

where $\delta_{\mathbf{z}}$ denotes the dirac mass on $\mathbf{z}$. By dividing by $p\left(\mathbf{x};\boldsymbol{\theta}^{\star}\right)$, we obtain

[TABLE]

As this inequality is true for every couple $(\mathbf{z},\mathbf{w})$, we have in particular:

[TABLE]

Noticing that $p\left(\mathbf{x};\boldsymbol{\theta}^{\star}\right)=\#\operatorname{Sym}(\boldsymbol{\theta}^{\star})p\left(\mathbf{x},\mathbf{z}^{\star},\mathbf{w}^{\star};\boldsymbol{\theta}^{\star}\right)(1+o_{p}(1))$, Theorem 5.1 therefore leads to the following bounds:

[TABLE]

Again, unless $\boldsymbol{\Theta}$ is constrained, $\widehat{\boldsymbol{\theta}}_{VAR}$ exhibits no symmetries with high probability and the same proof by contradiction as in section 6.1 gives the result.

$\square$

7 Conclusion

The Latent Block Model offers challenging theoretical questions. We solved under mild assumptions the consistency and asymptotic normality of the maximum likelihood and variational estimators for observations with conditional density belonging to a univariate exponential family, and for a balanced asymptotic rate between the number of rows $n$ and the number of columns $d$: $\log(d)/n\rightarrow 0$ and $\log(n)/d\rightarrow 0$ as $n$ and $d$ tend to infinity. Our results extend those of [2] for binary SBM not only by managing the double direction of LBM, but also by considering larger types of observations. That brought us to define specific concentration inequalities as large and moderate deviations concerning sub-exponential variables. Moreover, we dealt with specific cases of symmetry that were not taken into account as of now.

A specific framework of sparsity was studied by [2]. This is especially convenient for SBM, to model reasonable network settings: increasing sparsity (i.e. number of [math]) can be done directly by scaling the Bernoulli parameters $p_{k\ell}$ with a common factor that should decrease no faster than $\Omega(\log^{\delta}(n)/n)$, with $\delta>2$, to ensure consistency. This could also be considered for binary LBM. However this approach fails to model actual observations in the more general valued setting. The equivalent approach could be to consider the product of a Bernoulli variable with the actual observation value. Note however, than even without considering sparsity we recover essentially the same rate: in the sparse-SBM case, each node should be connected to $\Omega(\log^{\delta}(n))$ others to ensure consistency whereas in the dense-LBM case, each of the $n$-row should should be characterized by $\Omega(\log^{\delta}(n))$ columns (and vice-versa) to ensure consistency.

Alternative research direction could be to explore asymptotic settings where the numbers $n$ of rows and $d$ columns grow at very different rates. Other open question concern estimation of the number of row and column groups and settings where the number of groups increases with $n$ and $d$.

Appendix A Proofs of section 4

A.1 Proof of Proposition 4.5 (maximum of $G$ and $\tilde{\Lambda}$ in $\theta$)

Proof.

Define $\nu(x,\alpha)=x\alpha-\psi(\alpha)$. For $x$ fixed, $\nu(x,\alpha)$ is maximized at $\alpha=(\psi^{\prime})^{-1}(x)$. Manipulations yield

[TABLE]

which is maximized at $\alpha_{k\ell}=(\psi^{\prime})^{-1}(\widehat{x}_{k\ell}(\mathbf{z},\mathbf{w}))$. Similarly

[TABLE]

is maximized at $\alpha_{k\ell}=(\psi^{\prime})^{-1}(\bar{x}_{k\ell}(\mathbf{z},\mathbf{w}))$

$\square$

A.2 Proof of Proposition 4.6 (maximum of $G$ and $\tilde{\Lambda}$ in $(\theta,\mathbf{z},\mathbf{w})$)

Proof.

We condition on $(\mathbf{z}^{\star},\mathbf{w}^{\star})$ and prove Equation (4.2):

[TABLE]

If $(\mathbf{z}^{\star},\mathbf{w}^{\star})$ is regular, and for $n,d>2/c$, all the rows of ${I\!R}_{g}(\mathbf{z})$ and ${I\!R}_{m}(\mathbf{w})$ have at least one positive element and we can apply lemma D.1 (which is an adaptation for LBM of Lemma 3.2 of [2] for SBM) to characterize the maximum for $G$.

The maximality of $\tilde{\Lambda}(\mathbf{z}^{\star},\mathbf{w}^{\star})$ results from the fact that $\tilde{\Lambda}(\mathbf{z},\mathbf{w})=G(\bar{\boldsymbol{\alpha}}(\mathbf{z},\mathbf{w}),\mathbf{z},\mathbf{w})$ where $\bar{\boldsymbol{\alpha}}(\mathbf{z},\mathbf{w})$ is a particular value of $\boldsymbol{\alpha}$, $\tilde{\Lambda}$ is immediately maximum at $(\mathbf{z},\mathbf{w})\sim(\mathbf{z}^{\star},\mathbf{w}^{\star})$, and for those, we have $\bar{\boldsymbol{\alpha}}(\mathbf{z},\mathbf{w})\sim\boldsymbol{\alpha}^{\star}$.

The separation and local behavior of $G$ around $(\mathbf{z}^{\star},\mathbf{w}^{\star})$ is a direct consequence of the proposition 4.7.

$\square$

A.3 Proof of Proposition 4.7 (Local upper bound for $\tilde{\Lambda}$)

Proof.

We work conditionally on $(\mathbf{z}^{\star},\mathbf{w}^{\star})$. The principle of the proof relies on the extension of $\tilde{\Lambda}$ to a continuous subspace of $M_{g}([0,1])\times M_{m}([0,1])$, in which confusion matrices are naturally embedded. The regularity assumption allows us to work on a subspace that is bounded away from the borders of $M_{g}([0,1])\times M_{m}([0,1])$. The proof then proceeds by (1) computing the gradient of $\tilde{\Lambda}$ at and around its argmax and (2) using those gradients to control the local behavior of $\tilde{\Lambda}$ around its argmax. The local behavior allows in turn to show that $\tilde{\Lambda}$ is well-separated.

Note that $\tilde{\Lambda}$ only depends on $\mathbf{z}$ and $\mathbf{w}$ through ${I\!R}_{g}(\mathbf{z})$ and ${I\!R}_{m}(\mathbf{w})$. We can therefore extend it to matrices $(U,V)\in\mathcal{U}_{c}\times\mathcal{V}_{c}$ where $\mathcal{U}$ is the subset of matrices $\mathcal{M}_{g}([0,1])$ with each row sum higher than $c/2$ and $\mathcal{V}$ is a similar subset of $\mathcal{M}_{m}([0,1])$.

[TABLE]

where

[TABLE]

and $\mathbf{1}$ is the $g\times m$ matrix filled with $1$. Confusion matrices ${I\!R}_{g}(\mathbf{z})$ and ${I\!R}_{m}(\mathbf{w})$ satisfy ${I\!R}_{g}(\mathbf{z}){1\!I}={\widehat{\boldsymbol{\pi}}(\mathbf{z}^{\star})}$ and ${I\!R}_{m}(\mathbf{w}){1\!I}=\widehat{\boldsymbol{\rho}}(\mathbf{w}^{\star})$, with ${1\!I}=(1,\ldots,1){{}^{T}}$ a vector only containing $1$ values, and are obviously in $\mathcal{U}_{c}$ and $\mathcal{V}_{c}$ as soon as $(\mathbf{z}^{\star},\mathbf{w}^{\star})$ is $c/2$ regular.

The maps $f_{k,\ell}:(U,V)\mapsto KL(\alpha^{\star}_{k\ell},\bar{\alpha}_{k\ell}(U,V))$ are twice differentiable with second derivatives bounded over $\mathcal{U}_{c}\times\mathcal{V}_{c}$ and therefore so is $\tilde{\Lambda}(U,V)$. Tedious but straightforward computations show that the derivative of $\tilde{\Lambda}$ at $(D_{\pi},D_{\rho})\coloneqq(\operatorname{Diag}(\widehat{\boldsymbol{\pi}}(\mathbf{z}^{\star})),\operatorname{Diag}(\widehat{\boldsymbol{\rho}}(\mathbf{w}^{\star})))$ is:

[TABLE]

$A(\mathbf{w}^{\star})$ and $B(\mathbf{z}^{\star})$ are the matrix-derivative of $-\tilde{\Lambda}/nd$ at $(D_{\pi},D_{\rho})$. Since $(\mathbf{z}^{\star},\mathbf{w}^{\star})$ is $c/2$-regular and by definition of $\delta(\boldsymbol{\alpha}^{\star})$, $A(\mathbf{w}^{\star})_{kk^{\prime}}\geq c\delta(\boldsymbol{\alpha}^{\star})/2$ (resp. $B(\mathbf{w}^{\star})_{\ell\ell^{\prime}}\geq c\delta(\boldsymbol{\alpha}^{\star})/2$) if $k\neq k^{\prime}$ (resp. $\ell\neq\ell^{\prime}$) and $A(\mathbf{w}^{\star})_{kk}=0$ (resp. $B(\mathbf{z}^{\star})_{\ell\ell}=0$) for all $k$ (resp. $\ell$). By boundedness of the second derivative, there exists $C>0$ such that for all $(D_{\pi},D_{\rho})$ and all $(H,G)\in S(D_{\pi},D_{\rho},C)$, where the definition of the set of local assignments $S$ is extended to the subset of matrices, we have:

[TABLE]

Choose $U$ and $V$ in $(\mathcal{U}_{c}\times\mathcal{V}_{c})\cap S(D_{\pi},D_{\rho},C)$ satisfying $U{1\!I}=\boldsymbol{\pi}(\mathbf{z}^{\star})$ and $V{1\!I}=\boldsymbol{\rho}(\mathbf{w}^{\star})$. $U-D_{\pi}$ and $V-D_{\rho}$ have nonnegative off diagonal coefficients and negative diagonal coefficients. Furthermore, the coefficients of $U,V,D_{\pi},D_{\rho}$ sum up to $1$ and $\operatorname{Tr}(D_{\pi})=\operatorname{Tr}(D_{\rho})=1$. By Taylor expansion, there exists a couple $(H,G)$ also in $(\mathcal{U}_{c}\times\mathcal{V}_{c})\cap S(D_{\pi},D_{\rho},C)$ such that

[TABLE]

To conclude the proof, assume without loss of generality that $(\mathbf{z},\mathbf{w})\in S(\mathbf{z}^{\star},\mathbf{w}^{\star},C)$ achieves the $\|.\|_{0,\sim}$ norm (i.e. it is the closest to $(\mathbf{z}^{\star},\mathbf{w}^{\star})$ in its representative class). Then $(U,V)=({I\!R}_{g}(\mathbf{z}),{I\!R}_{m}(\mathbf{w}))$ is in $(\mathcal{U}_{c}\times\mathcal{V}_{c})\cap S(D_{\pi},D_{\rho},C)$ and satisfy $U{1\!I}=\boldsymbol{\pi}(\mathbf{z}^{\star})$ (resp. $V{1\!I}=\boldsymbol{\rho}(\mathbf{w}^{\star})$). We just need to note $n(1-\operatorname{Tr}({I\!R}_{g}(\mathbf{z})))=\|\mathbf{z}-\mathbf{z}^{\star}\|_{0,\sim}$ (resp. $d(1-\operatorname{Tr}({I\!R}_{m}(\mathbf{w})))=\|\mathbf{w}-\mathbf{w}^{\star}\|_{0,\sim}$) to end the proof.

$\square$

Appendix B Proofs of section 5

B.1 Proof of Proposition 5.5 (large deviation for $F_{nd}$)

Proof.

Conditionally upon $(\mathbf{z}^{\star},\mathbf{w}^{\star})$,

[TABLE]

uniformly in $\boldsymbol{\theta}$, where the $W_{kk^{\prime}\ell\ell^{\prime}}$ are independent and defined by:

[TABLE]

is the sum of $nd{I\!R}_{g}(\mathbf{z})_{kk^{\prime}}{I\!R}_{m}(\mathbf{w})_{\ell\ell^{\prime}}$ sub-exponential variables with parameters $(\bar{\sigma}^{2},1/\kappa)$ and is therefore itself sub-exponential with parameters $(nd{I\!R}_{g}(\mathbf{z})_{kk^{\prime}}{I\!R}_{m}(\mathbf{w})_{\ell\ell^{\prime}}\bar{\sigma}^{2},1/\kappa)$. According to Proposition C.4, $\mathbb{E}_{\boldsymbol{\theta}^{\star}}[Z|\mathbf{z}^{\star},\mathbf{w}^{\star}]\leq gm\operatorname{Diam}(\boldsymbol{\Theta})\sqrt{nd\bar{\sigma}^{2}}$ and $Z$ is sub-exponential with parameters $(nd\operatorname{Diam}(\boldsymbol{\Theta})^{2}(2\sqrt{2})^{2}\bar{\sigma}^{2},2\sqrt{2}\operatorname{Diam}(\boldsymbol{\Theta})/\kappa)$. In particular, for all $\varepsilon_{n,d}<\bar{\sigma}\kappa$

[TABLE]

We can then remove the conditioning and take a union bound to prove Equation (5.1).

$\square$

B.2 Proof of Proposition 5.6 (contribution of far away assignments)

Proof.

Conditionally on $(\mathbf{z}^{\star},\mathbf{w}^{\star})$, we know from proposition 4.6 that $\tilde{\Lambda}$ is maximal in $(\mathbf{z}^{\star},\mathbf{w}^{\star})$ and its equivalence class. Choose $0<t_{nd}$ decreasing to [math] but satisfying $t_{nd}^{2}\gg\frac{n+d}{nd}$. This is possible as $n\to 0$ and $d\to 0$.

According to equation 4.4, for all $(\mathbf{z},\mathbf{w})\notin S(\mathbf{z}^{\star},\mathbf{w}^{\star},C)$

[TABLE]

Now, according to Equation 4.3, for all $(\mathbf{z},\mathbf{w})\in S(\mathbf{z}^{\star},\mathbf{w}^{\star},C)\setminus S(\mathbf{z}^{\star},\mathbf{w}^{\star},t_{nd})$

[TABLE]

since either $\|\mathbf{z}-\mathbf{z}^{\star}\|_{0,\sim}\geq nt_{nd}$ or $\|\mathbf{w}-\mathbf{w}^{\star}\|_{0,\sim}\geq dt_{nd}$. Hence, for $n$ and $d$ large enough for all $(\mathbf{z},\mathbf{w})\notin S(\mathbf{z}^{\star},\mathbf{w}^{\star},t_{nd})$

[TABLE]

Set $\varepsilon_{nd}=\inf\left(\frac{c\delta(\boldsymbol{\alpha}^{\star})t_{nd}}{32\sqrt{2}\bar{\sigma}\operatorname{Diam}(\boldsymbol{\Theta})},\kappa\bar{\sigma}\right)$. By proposition 5.5, and with our choice of $\varepsilon_{nd}$, with probability higher than $1-\Delta_{nd}^{1}(\varepsilon_{nd})$,

[TABLE]

where the second line comes from inequality (B.1), the third from the global control studied in Proposition 5.5 and the definition of $\varepsilon_{nd}$, the fourth from the definition of $p(\mathbf{x},\mathbf{z}^{\star},\mathbf{w}^{\star};\boldsymbol{\theta}^{\star})$, the fifth from the bounds on $\boldsymbol{\pi}^{\star}$ and $\boldsymbol{\rho}^{\star}$ and the last from $t_{nd}\gg(n+d)/nd$.

In addition, we have $\varepsilon_{nd}^{2}\gg\frac{n+d}{nd}$ so that $\Delta_{nd}^{1}(\varepsilon_{nd})$ vanishes and:

[TABLE]

where the $o_{P}$ is uniform in probability over all $\boldsymbol{\theta}\in\boldsymbol{\Theta}$.

$\square$

B.3 Proof of Proposition 5.7 (local convergence $F_{nd}$)

Proof.

We work conditionally on $(\mathbf{z}^{\star},\mathbf{w}^{\star})\in\mathcal{Z}_{1}\times\mathcal{W}_{1}$. We assume with no loss of generality that $(\mathbf{z},\mathbf{w})$ is the representative of its class closest to $(\mathbf{z}^{\star},\mathbf{w}^{\star})$. Choose $\varepsilon\leq\kappa\underline{\sigma}^{2}$ small. Manipulation of $\Lambda$ and $\tilde{\Lambda}$ yield

[TABLE]

where $f(x)=x(\psi^{\prime})^{-1}(x)-\psi\circ(\psi^{\prime})^{-1}(x)$, $\widehat{x}_{k\ell}^{\star}=\widehat{x}_{k\ell}(\mathbf{z}^{\star},\mathbf{w}^{\star})$ and $\bar{x}_{k\ell}^{\star}=\psi^{\prime}(\alpha^{\star}_{k\ell})$. The function $f$ is twice differentiable on $\mathring{\mathcal{A}}$ with $f^{\prime}(x)=-(\psi^{\prime})^{-1}(x)$ and $f^{\prime\prime}(x)=-1/\psi^{\prime\prime}\circ(\psi^{\prime})^{-1}(x)$. $f^{\prime}$ (resp. $f^{\prime\prime}$) are bounded over $I=\psi^{\prime}([-M_{\alpha},M_{\alpha}])$ by $M_{\alpha}$ (resp. $1/\underline{\sigma}^{2}$).

Assignments $(\mathbf{z},\mathbf{w})$ belonging to $S(\mathbf{z}^{\star},\mathbf{w}^{\star},c/4)$ are also $c/4$-regular. According to Proposition C.2, $\widehat{x}_{k\ell}$ and $\bar{x}_{k\ell}$ are at distance at most $\varepsilon$ with probability higher than $1-2\exp\left(-\frac{ndc^{2}\varepsilon^{2}}{32(\bar{\sigma}^{2}+\kappa^{-1}\varepsilon)}\right)$, so that:

[TABLE]

By Proposition C.2, $(\widehat{x}_{k\ell}-\bar{x}_{k\ell})^{2}=\mathcal{O}_{P}(1/nd)$ where the $\mathcal{O}_{P}$ is uniform in $\mathbf{z},\mathbf{w}$ and does not depend on $\mathbf{z}^{\star},\mathbf{w}^{\star}$. Similarly,

[TABLE]

$\bar{x}_{k\ell}$ is a convex combination of the $S^{\star}_{k\ell}=\psi^{\prime}(\alpha^{\star}_{k\ell})$ therefore,

[TABLE]

Note that:

[TABLE]

and $\widehat{x}_{k\ell}-\bar{x}_{k\ell}=o_{P}(1)$. Therefore

[TABLE]

The remaining term writes

[TABLE]

According to Proposition C.3, this term is $o_{P}(\left(\frac{\|\mathbf{z}-\mathbf{z}^{\star}\|_{0,\sim}}{n}+\frac{\|\mathbf{w}-\mathbf{w}^{\star}\|_{0,\sim}}{d}\right)$ uniformly in $(\mathbf{z},\mathbf{w})$ and $(\mathbf{z}^{\star},\mathbf{w}^{\star})\in\Omega_{1}$ as soon $ne^{-ad}\to 0$ and $de^{-an}\to 0$ for all $a>0$, which is true under $(H_{4})$. It follows that:

[TABLE]

$\square$

B.4 Proof of Proposition 5.8 (contribution of local assignments)

Proof.

By Proposition 4.2, it is enough to prove that the sum is small compared to $p(\mathbf{z}^{\star},\mathbf{w}^{\star},\mathbf{x};\boldsymbol{\theta}^{\star})$ on $\Omega_{1}$. We work conditionally on $(\mathbf{z}^{\star},\mathbf{w}^{\star})\in\mathcal{Z}_{1}\times\mathcal{W}_{1}$. Choose $(\mathbf{z},\mathbf{w})$ in $S(\mathbf{z}^{\star},\mathbf{w}^{\star},\tilde{c})$. This set is non empty as soon as $\min(\tilde{c}n,\tilde{c}d)>1$.

[TABLE]

We can assume without loss of generality that $(\mathbf{z},\mathbf{w})$ is the representative closest to $(\mathbf{z}^{\star},\mathbf{w}^{\star})$ and denote $r_{1}=\|\mathbf{z}-\mathbf{z}^{\star}\|_{0}$ and $r_{2}=\|\mathbf{w}-\mathbf{w}^{\star}\|_{0}$. Then:

[TABLE]

where the first line comes from the definition of $\Lambda$, the second line from Proposition 4.7 and the fact that $\tilde{c}<C$ and the third from Proposition 5.7 and the fact that $\tilde{c}<c/4$. Thanks to corollary D.3, we also know that:

[TABLE]

There are at most ${n\choose r_{1}}{n\choose r_{2}}g^{r_{1}}m^{r_{2}}$ assignments $(\mathbf{z},\mathbf{w})$ at distance $r_{1}$ and $r_{2}$ of $(\mathbf{z}^{\star},\mathbf{w}^{\star})$ and each of them has at most $g^{g}m^{m}$ equivalent configurations. Therefore,

[TABLE]

where $a_{nd}=ne^{\log g+M_{c/4}-d\frac{c\delta(\boldsymbol{\alpha}^{\star})(1+o_{P}(1))}{4}}+de^{\log m+M_{c/4}-n\frac{c\delta(\boldsymbol{\alpha}^{\star})(1+o_{P}(1))}{4}}=o_{P}(1)$ as soon as $n\gg\log d$ and $d\gg\log n$.

$\square$

B.5 Proof of Proposition 5.9 (contribution of equivalent assignments)

Proof.

Choose $(s,t)$ permutations of $\{1,\dots,g\}$ and $\{1,\dots,m\}$ and assume that $\mathbf{z}=\mathbf{z}^{\star,s}$ and $\mathbf{w}=\mathbf{w}^{\star,t}$. Then $p(\mathbf{x},\mathbf{z},\mathbf{w};\boldsymbol{\theta})=p(\mathbf{x},\mathbf{z}^{\star,s},\mathbf{w}^{\star,t};\boldsymbol{\theta})=p(\mathbf{x},\mathbf{z}^{\star},\mathbf{w}^{\star};\boldsymbol{\theta}^{s,t})$. If furthermore $(s,t)\in\operatorname{Sym}(\boldsymbol{\theta})$, $\boldsymbol{\theta}^{s,t}=\boldsymbol{\theta}$ and immediately $p(\mathbf{x},\mathbf{z},\mathbf{w};\boldsymbol{\theta})=p(\mathbf{x},\mathbf{z}^{\star},\mathbf{w}^{\star};\boldsymbol{\theta})$. We can therefore partition the sum as

[TABLE]

The complete likelihood $p(\mathbf{x},\mathbf{z}^{\star},\mathbf{w}^{\star};\boldsymbol{\theta})$ is a unimodal function of $\boldsymbol{\theta}$ with mode located in $\widehat{\boldsymbol{\theta}}_{c}$. By consistency of $\widehat{\boldsymbol{\theta}}_{c}$, either $p(\mathbf{x},\mathbf{z}^{\star},\mathbf{w}^{\star};\boldsymbol{\theta})=o_{P}(p(\mathbf{x},\mathbf{z}^{\star},\mathbf{w}^{\star};\boldsymbol{\theta}^{\star}))$ or $p(\mathbf{x},\mathbf{z}^{\star},\mathbf{w}^{\star};\boldsymbol{\theta})=\mathcal{O}_{P}(p(\mathbf{x},\mathbf{z}^{\star},\mathbf{w}^{\star};\boldsymbol{\theta}^{\star}))$ when $\boldsymbol{\theta}$ is in a close neighborhood of $\boldsymbol{\theta}^{\star}$. In the latter case, any $\boldsymbol{\theta}^{\prime}\sim\boldsymbol{\theta}$ other than $\boldsymbol{\theta}$ is bounded away from $\boldsymbol{\theta}^{\star}$ and thus $p(\mathbf{x},\mathbf{z}^{\star},\mathbf{w}^{\star};\boldsymbol{\theta}^{\prime})=o_{P}(p(\mathbf{x},\mathbf{z}^{\star},\mathbf{w}^{\star};\boldsymbol{\theta}^{\star}))$. In summary,

[TABLE]

$\square$

o prove the Equation 6.2, first write

[TABLE]

that implies

[TABLE]

If $\boldsymbol{\theta}\approx\boldsymbol{\theta}^{\star}$ in the sense of [10], that is to say $\boldsymbol{\alpha}\approx\boldsymbol{\alpha}^{\star}$, we have

[TABLE]

Thus, for $\boldsymbol{\theta}\approx\boldsymbol{\theta}^{\star}$,

[TABLE]

When $\boldsymbol{\theta}\not{\!\!\approx}\boldsymbol{\theta}^{\star}$,

[TABLE]

Finally, we have $\widehat{\boldsymbol{\alpha}}_{var}\approx\boldsymbol{\alpha}^{\star}$ for $n$ and $d$ enough larges and with the same arguments as the proof of the proposition 6.1, we have $\widehat{\boldsymbol{\alpha}}\left(\mathbf{z}^{\star},\mathbf{w}^{\star}\right)-\widehat{\boldsymbol{\alpha}}_{var}^{s,t}=o_{P}\left(1\right)$.

Appendix C Concentration for sub-exponential variables

Proof of Lemma 4.2 A laisser? ou à mettre dans le texte?

Proof.

Each $z_{+k}$ is a sum of $n$ i.i.d Bernoulli r.v. with parameter $\pi_{k}\geq\pi_{\min}\geq c$. A simple Hoeffding bound shows that

[TABLE]

Similarly,

[TABLE]

The proposition follows from a union bound over $g$ values of $k$ and $m$ values of $\ell$.

$\square$

Concentration inequalities for sub-exponential variables play a key role: in particular Proposition C.4 for global convergence and Propositions C.2 and C.3 for local convergence. We present here some properties of sub-exponential variables ([14]), then derives the needed concentration inequalities.

Recall first that a random variable $X$ is sub-exponential with parameters $(\tau^{2},b)$ if for all $\lambda$ such that $|\lambda|\leq 1/b$,

[TABLE]

In particular, all distributions coming from a natural exponential family are sub-exponential. Sub-exponential variables satisfy a large deviation Bernstein-type inequality:

[TABLE]

So that

[TABLE]

C.1 Properties

The sub-exponential property is preserved by summation and multiplication.

• •

If $X$ is sub-exponential with parameters $(\tau^{2},b)$ and $\alpha\in\mathbb{R}$, then so is $\alpha X$ with parameters $(\alpha^{2}\tau^{2},\alpha b)$

• •

If the $X_{i}$, $i=1,\dots,n$ are sub-exponential with parameters $(\tau_{i}^{2},b_{i})$ and independent, then so is $X=X_{1}+\dots+X_{n}$ with parameters $(\sum_{i}\tau_{i}^{2},\max_{i}b_{i})$

Moreover, Lemma C.1 defines the sub-exponential property of the absolute value of a sub-exponential variable.

Lemma C.1**.**

If $X$ is a zero mean random variable, sub-exponential with parameters $(\sigma^{2},b)$, then $|X|$ is sub-exponential with parameters $(8\sigma^{2},2\sqrt{2}b)$.

Proof.

Denote $\mu=\mathbb{E}|X|$ and consider $Y=|X|-\mu$. Choose $\lambda$ such that $|\lambda|<(2\sqrt{2}b)^{-1}$. We need to bound $\mathbb{E}[e^{\lambda Y}]$. Note first that $\mathbb{E}[e^{\lambda Y}]\leq\mathbb{E}[e^{\lambda X}]+\mathbb{E}[e^{-\lambda X}]<+\infty$ is properly defined by sub-exponential property of $X$ and we have

[TABLE]

where we used the fact that $\mathbb{E}[Y]=0$. We know bound odd moments of $|\lambda Y|$.

[TABLE]

where we used first Cauchy-Schwarz and then the arithmetic-geometric mean inequality. The Taylor series expansion can thus be reduced to

[TABLE]

where we used the well-known inequality $\mathbb{E}[|X-\mathbb{E}[X]|^{k}]\leq 2^{k}\mathbb{E}[|X|^{k}]$ to substitute $2^{2k}\mathbb{E}[X^{2k}]$ to $\mathbb{E}[Y^{2k}]$.

$\square$

C.2 Concentration inequalities

Proposition C.2** (Maximum in $(\mathbf{z},\mathbf{w})$).**

Let $(\mathbf{z},\mathbf{w})$ be a configuration and $\widehat{x}_{k,\ell}(\mathbf{z},\mathbf{w})$ resp. $\bar{x}_{k\ell}(\mathbf{z},\mathbf{w})$ be defined in Equations (3.1) and (4.1). Under the assumptions of the section 2.2, for all $\varepsilon>0$

[TABLE]

Additionally, the suprema over all $c/2$-regular assignments satisfies:

[TABLE]

Note that equations C.3 and C.2 remain valid when replacing $c/2$ by any $\tilde{c}<c/2$.

Proof.

The random variables $X_{ij}$ are subexponential with parameters $(\bar{\sigma}^{2},1/\kappa)$. Conditionally to $(\mathbf{z}^{\star},\mathbf{w}^{\star})$, $z_{+k}w_{+\ell}(\widehat{x}_{k,\ell}-\bar{x}_{k\ell})$ is a sum of $z_{+k}w_{+\ell}$ centered subexponential random variables. By Bernstein’s inequality [11], we therefore have for all $t>0$

[TABLE]

In particular, if $t=ndx$,

[TABLE]

uniformly over $(\mathbf{z},\mathbf{w})$. Equation (C.3) then results from a union bound. Similarly,

[TABLE]

Where the last inequality comes from the fact that $c/2$-regular assignments satisfy $\widehat{\pi}_{k}(\mathbf{z})\widehat{\rho}_{\ell}(\mathbf{w})\geq c^{2}/4$. Equation (C.2) then results from a union bound over $\mathcal{Z}_{1}\times\mathcal{W}_{1}\subset\mathcal{Z}\times\mathcal{W}$.

$\square$

Proposition C.3** (Maximum in non equivalent $(\mathbf{z},\mathbf{w})$).**

Let $(\bar{\mathbf{z}},\bar{\mathbf{w}})$ be any configuration and $(\mathbf{z},\mathbf{w})$ the $\sim$-equivalent configuration that achieves $\|\mathbf{z}-\mathbf{z}^{\star}\|_{0}=\|\bar{\mathbf{z}}-\mathbf{z}^{\star}\|_{0,\sim}$ and $\|\mathbf{w}-\mathbf{w}^{\star}\|_{0}=\|\bar{\mathbf{w}}-\mathbf{w}^{\star}\|_{0,\sim}$, let $\widehat{x}_{k\ell}=\widehat{x}_{k,\ell}(\mathbf{z},\mathbf{w})$ (resp. $\bar{x}_{k\ell}(\mathbf{z},\mathbf{w})$) and $\widehat{x}_{k\ell}^{\star}=\widehat{x}_{k,\ell}(\mathbf{z}^{\star},\mathbf{w}^{\star})$ (resp. $\bar{x}_{k\ell}^{\star}=\bar{x}_{k\ell}(\mathbf{z}^{\star},\mathbf{w}^{\star})$ = $\psi^{\prime}(\alpha^{\star}_{k\ell})$) be as defined in Equations (3.1) and (4.1). Under the assumptions $(H_{1})$ to $(H_{3})$, for all $\varepsilon\leq\kappa\bar{\sigma}^{2}$,

[TABLE]

where $a_{nd}=nge^{-\frac{d\varepsilon^{2}}{\bar{\sigma}^{2}}}+mde^{-\frac{n\varepsilon^{2}}{\bar{\sigma}^{2}}}$.

Proof.

Denote $r_{1}=\|\mathbf{z}-\mathbf{z}^{\star}\|_{0}$ and $r_{2}=\|\mathbf{w}-\mathbf{w}^{\star}\|_{0}$. The numerator within the $\max$ in the fraction can be expanded to

[TABLE]

and is thus a sum of at most $N=nr_{2}+dr_{1}$ non-null centered sub-exponential random variables with parameters $(\bar{\sigma}^{2},1/\kappa)$. It is therefore centered sub-exponential with parameters $(N\bar{\sigma}^{2},1/\kappa)$. By Bernstein inequality, for all $\varepsilon\leq\kappa\bar{\sigma}^{2}$ we have

[TABLE]

There are at most ${n\choose r_{1}}g^{r_{1}}g^{g}$ $\mathbf{z}$ at $\|.\|_{0,\sim}$ distance $r_{1}$ of $\mathbf{z}^{\star}$ and ${d\choose r_{2}}m^{r_{2}}m^{m}$ $\mathbf{z}$ at $\|.\|_{0,\sim}$ distance $r_{2}$ of $\mathbf{w}^{\star}$. An union bound shows that:

[TABLE]

where $a_{nd}=nge^{-\frac{d\varepsilon^{2}}{\bar{\sigma}^{2}}}+mde^{-\frac{n\varepsilon^{2}}{\bar{\sigma}^{2}}}$.

$\square$

Proposition C.4** (concentration for sub-exponential).**

Let $X_{1},\dots,X_{n}$ be independent zero mean random variables, sub-exponential with parameters $(\sigma_{i}^{2},b_{i})$. Denote $V_{0}^{2}=\sum_{i}\sigma_{i}^{2}$ and $b=\max_{i}b_{i}$. Then the random variable $Z$ defined by:

[TABLE]

is also sub-exponential with parameters $(8M^{2}V_{0}^{2},2\sqrt{2}Mb)$. Moreover $\mathbb{E}[Z]\leq MV_{0}\sqrt{n}$ so that for all $t>0$,

[TABLE]

Proof.

Note first that $Z$ can be simplified to $Z=M\sum_{i}|X_{i}|$. We just need to bound bound $\mathbb{E}[Z]$. The rest of the proposition results from the fact that the $|X_{i}|$ are subexponential $(8\sigma_{i}^{2},2\sqrt{2}b_{i})$ by Lemma C.1 and standard properties of sums of independent rescaled subexponential variables.

[TABLE]

using Cauchy-Schwarz.

$\square$

Appendix D Technical lemmas

Lemma D.1 is the working horse for proving Proposition 4.6. Corollary D.3 is needed for Theorem 5.8 and Lemma D.2 is an intermediate result for Corollary D.3.

Lemma D.1**.**

Let $\eta$ and $\bar{\eta}$ be two matrices from $M_{g\times m}(\Theta)$ and $f:\Theta\times\Theta\to\mathbb{R_{+}}$ a positive function, $A$ a (squared) confusion matrix of size $g$ and $B$ a (squared) confusion matrix of size $m$. We denote $d_{k\ell k^{\prime}\ell^{\prime}}=f(\eta_{k\ell},\bar{\eta}_{k^{\prime}\ell^{\prime}})$. Assume that

• •

all the rows of $\eta$ are distinct;

• •

all the columns $\eta$ are distinct;

• •

$f(x,y)=0\Leftrightarrow x=y$;

• •

each row of $A$ has a non zero element;

• •

each row of $B$ has a non zero element;

and denote

[TABLE]

Then,

[TABLE]

Proof.

If $A$ and $B$ are the permutation matrices corresponding to the permutations $s$ et $t$: $A_{ij}=0$ if $i\neq s(j)$ and $B_{ij}=0$ if $i\neq t(j)$. As each row of $A$ contains a non zero element and as $A_{s(k)k}>0$ (resp. $B_{s(\ell)\ell}>0$) for all $k$ (resp. $\ell$), the following sum $\Sigma$ reduces to

[TABLE]

$\Sigma$ is null and sum of positive components, each component is null. However, all $A_{s(k)k}$ and $B_{t(\ell)\ell}$ are not null, so that for all $(k,\ell)$, $d_{s(k)t(\ell)k\ell}=0$ and $\bar{\eta}_{k\ell}=\eta_{s(k)t(\ell)}$.

Now, if $A$ is not a permutation matrix while $\Sigma=0$ (the same reasoning holds for $B$ or both). Then $A$ owns a column $k$ that contains two non zero elements, say $A_{k_{1}k}$ and $A_{k_{2}k}$. Let $\ell\in\{1\dots m\}$, there exists by assumption $\ell^{\prime}$ such that $B_{\ell\ell^{\prime}}\neq 0$. As $\Sigma=0$, both products $A_{k_{1}k}B_{\ell\ell^{\prime}}d_{k_{1}\ell k\ell^{\prime}}$ and $A_{k_{2}k}B_{\ell\ell^{\prime}}d_{k_{2}\ell k\ell^{\prime}}$ are zero.

[TABLE]

The previous equality is true for all $\ell$, thus rows $k_{1}$ and $k_{2}$ of $\eta$ are identical, and contradict the assumptions.

$\square$

Lemma D.2**.**

Let $\mathcal{Z}_{1}$ be the subset of $\mathcal{Z}$ of $c$-regular configurations, as defined in Definition 4.1. Let $\mathbb{S}^{g}=\{\boldsymbol{\pi}=(\pi_{1},\pi_{2},\dots,\pi_{g})\in[0,1]^{g}:\sum_{k=1}^{g}\pi_{k}=1\}$ be the $g$-dimensional simplex and denote $\mathbb{S}^{g}_{c}=\mathbb{S}^{g}\cap[c,1-c]^{g}$. Then there exists two positive constants $M_{c}$ and $M^{\prime}_{c}$ such that for all $\mathbf{z}$, $\mathbf{z}^{\star}$ in $\mathcal{Z}_{1}$ and all $\boldsymbol{\pi}\in\mathbb{S}_{c}^{g}$

[TABLE]

Proof.

Consider the entropy map $H:\mathbb{S}^{g}\to\mathbb{R}$ defined as $H(\boldsymbol{\pi})=-\sum_{k=1}^{g}\pi_{k}\log(\pi_{k})$. The gradient $\nabla H$ is uniformly bounded by $\frac{M_{c}}{2}=\log\frac{1-c}{c}$ in $\|.\|_{\infty}$-norm over $\mathbb{S}^{g}\cap[c,1-c]^{g}$. Therefore, for all $\boldsymbol{\pi}$, $\boldsymbol{\pi}^{\star}\in\mathbb{S}^{g}\cap[c,1-c]^{g}$, we have

[TABLE]

To prove the inequality, we remark that $\mathbf{z}\in\mathcal{Z}_{1}$ translates to $\widehat{\boldsymbol{\pi}}(\mathbf{z})\in\mathbb{S}^{g}\cap[c,1-c]^{g}$, that $\log p(\mathbf{z};\widehat{\boldsymbol{\pi}}(\mathbf{z}))-\log p(\mathbf{z}^{\star};\widehat{\boldsymbol{\pi}}(\mathbf{z}^{\star}))=n[H(\widehat{\boldsymbol{\pi}}(\mathbf{z}))-H(\widehat{\boldsymbol{\pi}}(\mathbf{z}^{\star}))]$ and finally that $\|\widehat{\boldsymbol{\pi}}(\mathbf{z})-\widehat{\boldsymbol{\pi}}(\mathbf{z}^{\star})\|_{1}\leq\frac{2}{n}\|\mathbf{z}-\mathbf{z}^{\star}\|_{0}$.

$\square$

Corollary D.3**.**

Let $\mathbf{z}^{\star}$ (resp. $\mathbf{w}^{\star}$) be $c/2$-regular and $\mathbf{z}$ (resp. $\mathbf{w}$) at $\|.\|_{0}$-distance $c/4$ of $\mathbf{z}^{\star}$ (resp. $\mathbf{w}^{\star}$). Then, for all $\boldsymbol{\theta}\in\boldsymbol{\Theta}$

[TABLE]

Proof.

Note then that:

[TABLE]

where the first inequality comes from the definition of $\widehat{\boldsymbol{\pi}}(\mathbf{z})$ and $\widehat{\boldsymbol{\rho}}(\mathbf{w})$ and the second from Lemma D.2 and the fact that $\mathbf{z}^{\star}$ and $\mathbf{z}$ (resp. $\mathbf{w}^{\star}$ and $\mathbf{w}$) are $c/4$-regular. Finally, local asymptotic normality of the MLE for multinomial proportions ensures that $\frac{p(\mathbf{z}^{\star},\mathbf{w}^{\star};\widehat{\boldsymbol{\pi}}(\mathbf{z}^{\star}),\widehat{\boldsymbol{\rho}}(\mathbf{w}^{\star}))}{p(\mathbf{z}^{\star},\mathbf{w}^{\star};\boldsymbol{\pi}^{\star},\boldsymbol{\rho}^{\star})}=\mathcal{O}_{P}(1)$.

$\square$