Exact Recovery with Symmetries for the Doubly-Stochastic Relaxation

TL;DR

This paper analyzes the conditions under which convex relaxations for graph matching with symmetries are exact, focusing on the role of symmetry groups and providing algorithms for recovering graph isomorphisms.

Contribution

It characterizes when convex relaxations are exact for symmetric graph matching problems and introduces algorithms for retrieving isomorphisms in these cases.

Findings

Convex exactness depends on the symmetry group of the graphs.

For reflective symmetry groups with at least one full orbit, convex exactness holds almost everywhere.

The proposed algorithms effectively retrieve isomorphisms when convex exactness is satisfied.

Abstract

Graph matching or quadratic assignment, is the problem of labeling the vertices of two graphs so that they are as similar as possible. A common method for approximately solving the NP-hard graph matching problem is relaxing it to a convex optimization problem over the set of doubly stochastic (DS) matrices. Recent analysis has shown that for almost all pairs of isomorphic and asymmetric graphs, the DS relaxation succeeds in correctly retrieving the isomorphism between the graphs. Our goal in this paper is to analyze the case of symmetric isomorphic graphs. This goal is motivated by shape matching applications where the graphs of interest usually have reflective symmetry. For symmetric problems the graph matching problem has multiple isomorphisms and so convex relaxations admit all convex combinations of these isomorphisms as viable solutions. If the convex relaxation does not admit any additional superfluous solution we say that it is convex exact. In this case there are tractable algorithms to retrieve an isomorphism from the convex relaxation. We show that convex exactness depends strongly on the symmetry group of the graphs; For a fixed symmetry group $G$, either the DS relaxation will be convex exact for almost all pairs of isomorphic graphs with symmetry group $G$, or the DS relaxation will fail for all such pairs. We show that for reflective groups with at least one full orbit convex exactness holds almost everywhere, and provide some simple examples of non-reflective symmetry groups for which convex exactness always fails. When convex exactness holds, the isomorphisms of the graphs are the extreme points of the convex solution set. We suggest an efficient algorithm for retrieving an isomorphism in this case. We also show that the "convex to concave" projection method will also retrieve an isomorphism in this case.

Equations

$$\underset{P\in {\Pi }_n}{\min }E(P)={\Vert AP-PB\Vert }_F.$$

$$A_{ij}=d_A({\mathbf{a}}_i,{\mathbf{a}}_j),B_{ij}=d_B({\mathbf{b}}_i,{\mathbf{b}}_j)$$

$$\text{DS}=\{S|S\text{1}=\text{1},{\text{1}}^{T}S={\text{1}}^T,S\ge 0\},$$

$$\underset{S\in \text{DS}}{\min }E(S)={\Vert AS-SB\Vert }_F.$$

$${\mathrm{ISO}}_{\mathrm{conv}}(A,B)=\{S\in \text{DS}|AS=SB\}.$$

$$\mathrm{ISO}(A,B)\subseteq {\mathrm{ISO}}_{\mathrm{conv}}(A,B)$$

$${\mathrm{ISO}}_{\mathrm{conv}}(A,B)=\mathrm{ISO}(A,B).$$

$$S\mapsto S{P}^T$$

$$\mathrm{Aut}(A)=\mathrm{ISO}(A,A),{\mathrm{Aut}}_{\mathrm{conv}}(A)={\mathrm{ISO}}_{\mathrm{conv}}(A,A),\text{DS}(A)=\text{DS}(A,A).$$

$$\mathrm{conv}\mathrm{Aut}(A)\subseteq {\mathrm{Aut}}_{\mathrm{conv}}(A).$$

$$\mathrm{conv}\mathrm{ISO}(A,B)={\mathrm{ISO}}_{\mathrm{conv}}(A,B).$$

$$\mathcal{A}(G)=\{A\in {\mathcal{S}}^n|\mathrm{Aut}(A)=G\}.$$

$${\mathbf{a}}_1\mapsto {\mathbf{a}}_2,{\mathbf{a}}_2\mapsto {\mathbf{a}}_3,{\mathbf{a}}_3\mapsto {\mathbf{a}}_1.$$

$$A_{11}=A_{22}=A_{33}\text{ and }{A}_{12}=A_{23}=A_{31}.$$

$$\mathcal{V}(G)=\{A\in {\mathcal{S}}^n|A=P^{T}AP\text{ for all }P\in G\}.$$

$$\mathcal{A}(G)=\mathcal{V}(G)\setminus \bigcup _{G⊊H\le {\Pi }_n}\mathcal{V}(H).$$

$$S_c=\frac{1}{|G|}\sum _{P\in G}P,$$

$$\mathcal{N}(G)=\{T\in {\mathbb{R}}^{n\times n}|T_{ij}=0\text{ if }{P}_{ij}=0\text{ for all }P\in G\}.$$

$$s(A)=A\text{1}.$$

$$Ss=SA\text{1}=AS\text{1}=A\text{1}=s.$$

$${\Vert s\Vert }^2=⟨Ss,s⟩=\sum _k{\theta }_k⟨P(k)s,s⟩{\le }^{(\ast )}\sum _k{\theta }_k\Vert P(k)\Vert {\Vert s\Vert }^2={\Vert s\Vert }^2.$$

$${\mathcal{V}}_{i,j}(G)=\{A\in \mathcal{V}(G)|s_i(A)=s_j(A)\}.$$

$${\bar{A}}_{ij}=r|I_r{|}^{-1}.$$

$$A_0=\left[\begin{array}{ccc}6& 1& 2\\ 1& 5& 3\\ 2& 3& 4\end{array}\right]$$

$$\underset{S\in \text{DS}}{\min }{\Vert S{A}_0-A_{0}S\Vert }_F^2+\sum _{ij}{S}_{ij}\Vert L(i)-L(j)\Vert$$

$$A_{ij}{S}_{jj}=S_{ii}{A}_{ij},\text{ for all }1\le i,j\le k.$$

$$A_{rj}{S}_{jj}=S_{rr}{A}_{rj},\text{ for all }j$$

$$S\mapsto S_{rr}$$

$$H=\{P_{rr}|P\in \mathrm{Aut}(A)\}$$

$${\mathrm{Aut}}_{\mathrm{conv}}(A)\subseteq \mathrm{conv}H$$

Full text

For symmetric problems the graph matching problem has multiple isomorphisms and so convex relaxations admit all convex combinations of these isomorphisms as viable solutions. If the convex relaxation does not admit any additional superfluous solution we say that it is convex exact.

We show that convex exactness depends strongly on the symmetry group of the graphs; For a fixed symmetry group $G$, either the DS relaxation will be convex exact for almost all pairs of isomorphic graphs with symmetry group $G$, or the DS relaxation will fail for all such pairs. We show that for reflective groups with at least one full orbit convex exactness holds almost everywhere, and provide some simple examples of non-reflective symmetry groups for which convex exactness always fails.

When convex exactness holds, the isomorphisms of the graphs are the extreme points of the convex solution set. We suggest an efficient algorithm for retrieving an isomorphism in this case. We also show that the ”convex to concave” projection method will also retrieve an isomorphism in this case, and show experimentally that this projection method as well as the standard Euclidean projection will succeed in retrieving an isomorphism for near isomorphic graphs as well.

In certain cases it is sufficient to find the centroid of the set of isomorphisms, which gives a ”fuzzy encoding” of the symmetries of the shape. We show that for any symmetry group $G$, the centroid solution can be recovered efficiently for almost all pairs of isomorphic graphs with symmetry group $G$. Additionally we show that for such isomorphic graphs interior-point solvers will generally return the centroid solution.

1 Introduction

Graph matching and graph isomorphism are classical problems in computer science. In this paper we will use the term graph for a pair $(\mathbf{a},A)$, where $\mathbf{a}=(\mathbf{a}_{1},\ldots,\mathbf{a}_{n})$ are the vertices of the graph, and $A$ is a symmetric matrix encoding the relationship between the vertices. We will also sometimes refer to $A$ alone as a graph. An isomorphism between graphs $(\mathbf{a},A)$ and $(\mathbf{b},B)$ is a relabeling of the vertices of $B$ so that $A$ and the relabeled $B$ are identical. The graphs $A$ and $B$ are isomorphic if there is an isomorphism between them. In matrix notation, an isomorphism is a permutation matrix $P$ such that $A=PBP^{T}$ or equivalently $AP=PB$. The problem of deciding whether two graphs are isomorphic is known as the Graph isomorphism problem (GI). It is not known to be in P, but is also not known to be NP-hard. Recently [Babai, 2016] provided a quasi-polynomial time algorithm for GI. While no polynomial algorithm for the general GI problem is known, there are many families of graphs for which GI can be solved in polynomial time. One example which is relevant for this work is graphs with simple spectrum, or more generally bounded eigenvalue multiplicity [Babai et al., 1982].

The graph matching problem is the problem of determining how close two graphs are to being isomorphic by minimizing the graph matching energy over the set of permutation matrices which we denote by $\Pi_{n}$:

[TABLE]

This optimization problem is also often referred to as the Koopmans-Beckmann quadratic assignment problem, and is usually phrased as the equivalent problem of maximizing $\mathrm{tr}APB^{T}P^{T}$. In contrast to GI whose computational status is not fully known, global minimization of quadratic assignment, and even approximation to within a constant factor, is known to be NP-hard [Sahni and Gonzalez, 1976].

Graph matching problems have found many applications. See for example [Conte et al., 2004] for a survey on applications of graph matching for pattern recognition. Our work is motivated by shape matching applications: Shape matching is the problem of measuring how similar two given surfaces $\mathcal{S}_{A},\mathcal{S}_{B}$ are. The notion of similarity between shapes is required to be invariant to shape preserving deformations such as rigid transformations for rigid objects (e.g., chairs), and deformations which preserve geodesic distances for non-rigid objects (e.g., humans). Accordingly shape matching problems are often modeled (e.g., [Mémoli, 2007, Mémoli, 2011, Solomon et al., 2016]) as the problem of finding a mapping between two surfaces $\mathcal{S}_{A},\mathcal{S}_{B}$ so that they are as isometric as possible. The metric on the shapes is typically either the extrinsic Euclidean metric for rigid shapes, or the intrinsic geodesic metric for non-rigid shapes.

Finding near-isometries between shapes can be phrased as a graph matching problem by selecting a finite sampling of the shapes to obtain vertices $\mathbf{a},\mathbf{b}$ on the two shapes, and taking $A,B$ to be the distance matrices defined by the distances on the shapes, that is

[TABLE]

In this setting an isomorphism between $A$ and $B$ corresponds to an isometry between the sampled metric spaces.

In this work we will focus on symmetric graphs, which are very relevant for shape matching applications since most natural shapes have intrinsic symmetries- that is, intrinsic isometries from the shape to itself other than the trivial identity mapping. Figure 1 shows some representative shapes from the [Giorgi et al., 2007] shape matching dataset. Typically natural shapes have a symmetry group with only two elements (bilateral symmetry) as in the left hand side of Figure 1, but there are interesting examples with larger symmetry groups as in the right hand side of Figure 1.

The doubly-stochastic relaxation

In this paper we focus on analyzing the doubly-stochastic (DS) relaxation for graph matching. For a survey on other convex relaxation and combinatorial methods which have been proposed to achieve good solutions for quadratic assignment see [Loiola et al., 2007].

The doubly stochastic (DS) relaxation replaces the NP hard graph matching problem with a tractable optimization problem by relaxing the combinatorial set of permutations to its convex hull of doubly stochastic matrices:

[TABLE]

which leads to a convex quadratic program known as the DS relaxation:

[TABLE]

We will refer to this optimization problem as $\text{DS}(A,B)$. Since the DS  relaxation minimizes $E(\cdot)$ over a larger domain, its minimum value is a lower bound for the minimal value of the graph matching problem. As can be expected due to the hardness of the problem, the DS  relaxation does not generally return the global minimum or minimizer of (1) [Lyzinski et al., 2016]. In particular, [Scheinerman and Ullman, 2011] characterizes all cases in which the minimum of (2) is zero even when the graphs are not isomorphic.

We will be interested in the case where $A$ and $B$ are isomorphic. Note that in this case the global minimum of (2) is zero and thus coincides with the global minimum of (1). The interesting question is whether the DS relaxation succeeds in returning a minimizer which is a permutation. Clearly we do not expect this will be the case for all graphs since this would provide us with a polynomial time algorithm to solve GI. On the other hand, since there are many families of graphs for which GI is tractable, we can hope that for many instances the DS relaxation will be successful in returning a permutation solution. The recent works of [Aflalo et al., 2015, Fiori and Sapiro, 2015] show that indeed this is the case. To state their results we introduce some notation:

Let us denote the set of isomorphisms of $A,B$ by $\mathrm{ISO}(A,B)$. We will say that $S\in\text{DS}$ is a convex isomorphism if it is a member of the set

[TABLE]

The inclusion

[TABLE]

is obvious. However it is possible that the DS relaxation will contain additional minimizers. We will say that $\text{DS}(A,B)$ is exact when this possibility does not occur and

[TABLE]

We note that the exactness property depends only on $A$: An isomorphism $P\in\mathrm{ISO}(A,B)$ defines a linear bijection

[TABLE]

from $\mathrm{ISO}_{\mathrm{conv}}(A,B)$ to $\mathrm{ISO}_{\mathrm{conv}}(A,A)$ and from $\mathrm{ISO}(A,B)$ to $\mathrm{ISO}(A,A)$. Accordingly if $A,B$ are isomorphic, $\text{DS}(A,B)$ is exact if and only if $\text{DS}(A,A)$ is exact. We will refer to (convex) isomorphisms in the case $A=B$ as (convex) automorphisms. We also denote:

[TABLE]

We say that $A$ is an asymmetric graph if the identity matrix is its only automorphism. Otherwise we say that $A$ is a symmetric graph. A necessary condition for exactness of $\text{DS}(A)$ is that $A$ is asymmetric. This is because if $A$ has several automorphisms then due to the inclusion (3) and the convexity of $\mathrm{Aut}_{\mathrm{conv}}(A)$

[TABLE]

Thus, while $A$ has a finite number of automorphisms, it has an infinite number of convex automorphisms. Even when $A$ is asymmetric, exactness does not always occur. A simple counter example will be discussed in Section 2. However, [Aflalo et al., 2015] showed that for asymmetric $A$ satisfying certain weak conditions exactness will hold. Their result was later shown to hold with even weaker conditions in [Fiori and Sapiro, 2015].

Convex exactness

Our goal in this paper is to show that for certain kinds of symmetry groups the DS relaxation can still be successfully applied, by defining a suitable notion of convex exactness. A similar goal has recently been achieved by [Dym and Lipman, 2016] for a semi-definite programming relaxation of the Procrustes matching problem.

We say that $\text{DS}(A)$ is convex exact if equality holds in (4), or equivalently if for any $B$ isomorphic to $A$,

[TABLE]

Note that for asymmetric graphs, convex exactness and exactness coincide. When convex exactness holds an isomorphism can be extracted in a tractable manner as we will discuss in Section 6.

For every permutation subgroup $G\leq\Pi_{n}$ we define

[TABLE]

In the asymmetric case $G=\{I_{n}\}$ we know that there are $A\in{\mathcal{A}}(G)$ such that $\text{DS}(A)$ is not (convex) exact, but also that (convex) exactness often does hold for asymmetric graphs. Our goal is to give a more precise notion of this claim by showing that for almost all asymmetric graphs (convex) exactness holds. More importantly, we would like to find non-trivial groups $G$ for which $\text{DS}(A)$ will be convex exact for almost every $A\in{\mathcal{A}}(G)$. To do so we must first define a natural measure $\mu_{G}$ on ${\mathcal{A}}(G)$.

We will assume that ${\mathcal{A}}(G)$ is non-empty. Permutation groups $G\leq\Pi_{n}$, for which ${\mathcal{A}}(G)$ is empty do exist. A simple example is the cyclic group $G\leq\Pi_{3}$ generated by the permutation

[TABLE]

Any $A\in{\mathcal{A}}(G)$ satisfies

[TABLE]

Thus, all diagonal elements of $A$ are identical and all off-diagonal elements of $A$ are identical as well since $A=A^{T}$. It follows that $\mathrm{ISO}(A)=\Pi_{3}$. If ${\mathcal{A}}(G)$ is non-empty we say that $G$ is a symmetry group.

For a symmetry group $G$ we consider the vector space

[TABLE]

Since $\mathcal{V}(G)$ is a vector space of some dimension $d$ it has a natural notion of measure- the $d$ dimensional Hausdorff measure on $\mathbb{R}^{n\times n}$ restricted to $\mathcal{V}(G)$, or equivalently the push forward of the Lebesgue measure on $\mathbb{R}^{d}$ to $\mathcal{V}(G)$ via a linear isometry between the two spaces. We denote this measure by $\mu_{G}$. Note that

[TABLE]

Since by assumption ${\mathcal{A}}(G)$ is non-empty it follows that all the $\mathcal{V}(H)$ are strict subspaces of $\mathcal{V}(G)$ and therefore the complement of ${\mathcal{A}}(G)$ in $\mathcal{V}(G)$ has measure zero. Thus $\mu_{G}$ is a natural choice for a measure on ${\mathcal{A}}(G)$. We will say that a property is generic, or that it holds for almost every $A\in{\mathcal{A}}(G)$, if it holds for $\mu_{G}$ almost every $A\in{\mathcal{A}}(G)$.

We can now state our main results:

1.1 Main results

Reflective groups

We show that convex exactness is a generic property for groups $G$ fulfilling the following two conditions:

Definition 1**.**

We say that $G\leq\Pi_{n}$ is a reflection group if $P^{2}=I_{n}$ for all $P\in G$.

Any group $G\leq\Pi_{n}$ defines an action $(\sigma,\mathbf{a}_{j})\mapsto\mathbf{a}_{\sigma(j)}$ on the set of vertices $\mathbf{a}$. We denote the orbit of $\mathbf{a}_{j}$ by $[\mathbf{a}_{j}]$. In general we have that $|[\mathbf{a}_{j}]|\leq|G|$.

Definition 2**.**

We say that $G$ has a full orbit if it has an orbit of length $|G|$.

In shape matching applications the full orbit assumption is typically fulfilled; an orbit $[\mathbf{a}_{i}]$ will be full unless $\mathbf{a}_{i}$ is on a symmetry axis of the shape. Under the full orbit and reflection group assumption, we prove:

Theorem 1**.**

Assume $G\leq\Pi_{n}$ is a reflection group with a full orbit. Then the DS relaxation is convex exact with respect to almost all $A\in{\mathcal{A}}(G)$.

As a result we obtain that convex exactness is a generic property for the simplest but, in the context of shape matching applications, most important, symmetry groups:

Corollary 1**.**

If $G\cong\mathbb{Z}_{2}$ then the DS relaxation is convex exact with respect to almost all $A\in{\mathcal{A}}(G)$.

General groups

For general groups we provide a ”zero-one probability” result:

Theorem 2**.**

For any symmetry group $G\leq\Pi_{n}$ one of the following holds:

The DS relaxation is convex exact with respect to almost every $A\in{\mathcal{A}}(G)$. 2. 2.

The DS relaxation is not convex exact for any $A\in{\mathcal{A}}(G)$.

The proof of Theorem 2 is constructive in the sense that it enables checking which of the two mutually exclusive alternatives described in the theorem hold for a given symmetry group $G$. By using this strategy we can establish that there are quite simple non-reflective symmetry groups for which convex exactness fails. Figure 2 shows nine groups $G_{i}$ represented by nine shapes whose symmetry group is $G_{i}$. For the first three groups (a)-(c) we found that convex exactness does not hold for any $A\in{\mathcal{A}}(G)$, while for the remaining groups convex exactness does hold for almost all $A\in{\mathcal{A}}(G)$. Note that all groups in the first column are isomorphic to $\mathbb{Z}_{3}$, all groups in the second column are isomorphic to $\mathbb{Z}_{4}$, and all groups in the last column are isomorphic to the dihedral group $D_{4}$. Thus we see that while convex exactness is a generic property for any $G$ isomorphic to $\mathbb{Z}_{2}$, in general different permutation groups can behave very differently with respect to the DS relaxation even if they are isomorphic in the sense of group theory.

Additional results: Permutation solutions and centroid solution

Since for symmetric problems the DS relaxation has an infinite number of convex isomorphisms, the question of achieving an ”interesting” convex isomorphism arises. Naturally we would like to achieve a convex isomorphism which is a permutation. In the case of convex exactness this reduces to the problem of finding an extreme point (a ”corner”) of the set of convex isomorphisms, which is known to be a tractable problem. In Section 6 we describe two known methods to obtain extreme points. Additionally we provide a much faster algorithm for achieving all isomorphisms between $A$ and $B$. This algorithm is valid for almost all graphs whose symmetry group $G$ satisfy the conditions of Theorem 1.

A disadvantage of the methods mentioned above for finding permutation solutions is that they are constructed for perfectly isomorphic problems and are not suited for near isomorphic problems and are not used in practice. Instead, permutations are typically obtained using the $L_{2}$ projection or the more accurate, but more expensive, ”convex to concave” projection. We prove that when convex exactness holds, the convex to concave projection is able to return an isomorphism. For symmetric problems with a small amount of noise, we show experimentally that both projection methods are generally able to retrieve an isomorphism, and the convex to concave method is often able to retrieve an isomorphism for higher noise levels as well.

An alternative ”interesting” convex isomorphism which is easier to find than ”corners” is the ”centroid” of the set of isomorphisms:

[TABLE]

where $G$ is the set of isomorphisms between $A$ and $B$. As advocated in [Solomon et al., 2012], finding $S_{c}$ in the case of symmetric problems can potentially be useful as it gives an ”encoding” of all isomorphisms of $A,B$. In Section 5 we show that the centroid solution is easier to find than corner solutions: In fact, for any symmetry group $G$, and almost every pair of isomorphic graphs $A,B\in\mathrm{ISO}(G)$, the centroid solution can be achieved (almost always) for any symmetry group. Additionally we show that for such $A,B$ penalty based optimization methods will converge to $S_{c}$ when solving $\text{DS}(A,B)$.

An illustration of the centroid solution is shown in Figure 3, for the problem of mapping a cylinder to itself, using as $A=B$ the Euclidean distance matrix of the cylinder. The right part of the figure shows the cylinder, colored so that points in the same orbit of the symmetry group of the cylinder share the same color. The left part of the figure shows the matrix $S_{c}$. Each yellow square in the left figure is a submatrix whose indices correspond to a circular section of the cylinder. It can be seen that the centroid solution assigns each point of the cylinder with equal probability to any other point in its orbit. We note that the centroid solution always has this property. Therefore different symmetry groups which have identical orbits will have the same centroid solution, and so the symmetry group cannot generally be reconstructed from the centroid solution.

The remainder of the paper is organized as follows: In Section 2 we define the notion of weak exactness which will be useful for the proofs presented later on. In Section 3 we prove convex exactness for reflective groups (Theorem 1). In Section 4 we prove our ”zero-one probability result” (Theorem 2) and explain how to check which of the two alternatives described in the theorem apply for a given group. In Section 5 we discuss the issue of retrieving the centroid solutions and finally in Section 6 we discuss the issue of retrieving isomorphisms in the case that convex exactness holds.

2 Weak exactness

An important tool for the proofs we present later on is the concept of weak exactness which we will now define: For any set $G$ of permutation matrices we define

[TABLE]

We say that $\text{DS}(A)$ is weakly exact if all convex automorphisms of $A$ are in $\mathcal{N}(\mathrm{Aut}(A))$. Less formally, this means that the $i,j$ coordinate of a convex automorphism $S$ can be positive only if there is an automorphism taking $\mathbf{a}_{i}$ to $\mathbf{a}_{j}$. If $\text{DS}(A)$ is weakly exact and $B$ is isomorphic to $A$ then all convex isomorphisms of $A,B$ are in $\mathcal{N}(\mathrm{ISO}(A,B))$. Weak exactness is guaranteed with full probability for any symmetry group $G$. We show this using the vector

[TABLE]

The vector $s(A)$ is invariant under automorphisms, meaning that if $\mathbf{a}_{j}\in[\mathbf{a}_{i}]$ then $s_{i}(A)=s_{j}(A)$. We say that $s(A)$ is discriminative if for any $i,j$ such that $\mathbf{a}_{j}\not\in[\mathbf{a}_{i}]$, we have $s_{i}(A)\neq s_{j}(A)$. We prove

Theorem 3**.**

Let $G$ be any symmetry group. Then

If $s(A)$ is discriminative then $\text{DS}(A)$ is weakly exact. 2. 2.

For almost every $A\in{\mathcal{A}}(G)$, the vector $s(A)$ is discriminative.

We prove Theorem 3. We first prove that if $s(A)$ is discriminative then $DS(A)$ is weakly exact. If $S$ is a convex isomorphism, then $s=s(A)$ is fixed by $S$ because

[TABLE]

Write $S$ as a convex combination of permutations $S=\sum_{k}\theta_{k}P(k)$. Using the fact that the operator norm of a permutation is one and the Cauchy-Schwartz inequality we obtain

[TABLE]

so $(*)$ is an equality, implying that $P(k)s=s$ for all $k$. Now if $\mathbf{a}_{j}\not\in[\mathbf{a}_{i}]$ then $P_{ij}(k)=0$ for all $k$ and therefore $S_{ij}=0$. Thus we have proven that $\text{DS}(A)$ is weakly exact when $s(A)$ is discriminative.

We now show that discriminativeness is a generic property. It is sufficient to show that for almost every $A\in\mathcal{V}(G)$ the claim holds since ${\mathcal{A}}(G)$ is a subset of $\mathcal{V}(G)$. Note that $s(A)$ is discriminative unless there are some $(i,j)$ such that $\mathbf{a}_{j}$ is not in $[\mathbf{a}_{i}]$ but $A$ is in the vector space

[TABLE]

Thus it is sufficient to show that all these spaces are strict subspaces of $\mathcal{V}(G)$, which we accomplish by finding a member $\bar{A}\in\mathcal{V}(G)$ for which $s(\bar{A})$ is discriminative.

To construct $\bar{A}$ let $I_{1},I_{2},\ldots,I_{k}$ be the partition of the vertices $\mathbf{a}$ induced by the action of $G$. For all $r\leq k$ and $\mathbf{a}_{i},\mathbf{a}_{j}\in I_{r}$ we set

[TABLE]

If $\mathbf{a}_{j}\not\in[\mathbf{a}_{i}]$ we set $\bar{A}_{ij}=0$. The constructed graph $\bar{A}$ is a member of $\mathcal{V}(G)$, and the vector $s(\bar{A})$ is discriminative since for all $r\leq k$ and $i\in I_{r}$ we have $s_{i}(\bar{A})=r$. This concludes the proof of Theorem 3.

Counter example

While weak exactness is guaranteed almost everywhere, it can still fail in very simple examples. Such examples can be constructed using the fact that if 1 is an eigenvector of $A$ then $\frac{1}{n}\mathrm{\textbf{1}}^{T}\mathrm{\textbf{1}}$ is always a valid convex automorphism. For example the graph

[TABLE]

is asymmetric, but satisfies $A_{0}\mathrm{\textbf{1}}=\lambda\mathrm{\textbf{1}}$ for $\lambda=9$ and so $\frac{1}{n}\mathrm{\textbf{1}}^{T}\mathrm{\textbf{1}}$ is a convex isomorphism, and so weak exactness, and certainly exactness, does not hold.

One method for overcoming such counter examples is adding a linear term to the graph matching energy penalizing for correspondences which do not respect isomorphism-invariants. For example, For each vertex $\mathbf{a}_{i}$ we can define $L(i)$ to be the sorted values of the $i$-th row of the graph. Clearly if $\mathbf{a}_{j}\in[\mathbf{a}_{i}]$ then $L(i)=L(j)$ so $L$ is an isomorphism invariant. Since in our example $L(i),i=1,2,3$ are all distinct, the only zero-energy solution of the modified relaxation

[TABLE]

is the identity matrix.

3 Convex exactness for reflective groups

Our goal in this section is proving convex exactness holds generically for reflective groups with a full orbit (Theorem 1). We break up the proof of the theorem into two parts: The first part establishes sufficient conditions which guarantee exact recovery, and the second part proves these sufficient conditions hold generically if $G$ is reflective and has a full orbit.

Fix some $G$ and $A\in{\mathcal{A}}(G)$. As in the previous section let $I_{1},I_{2},\ldots,I_{k}$ be the partition of the vertices $\mathbf{a}$ induced by the action of $G$, and denote $n_{j}=|I_{j}|$. Let $S$ be a convex automorphism of $A$. Denote by $S_{ij}$ and $A_{ij}$ the submatrices of $S,A$ corresponding to the indices $I_{i}\times I_{j}$. If $A$ is weakly exact then $S_{ij}=0$ whenever $i\neq j$ and so the equation $AS=SA$ takes the form

[TABLE]

3.1 Sufficient conditions for convex exactness

Proposition 3.1**.**

Let $A$ be a graph. If $\exists r,1\leq r\leq k$ such that

The vector $s(A)$ is discriminative. 2. 2.

$\text{rank}(A_{rj})=n_{j}$* for all $j\neq r$.* 3. 3.

$A_{rr}$* has simple spectrum.*

then $\text{DS}(A)$ is convex exact.

Proof of Proposition 3.1.

The first condition guarantees weak exactness, and thus that all convex isomorphisms will satisfy (8). Setting $i=r$ in this equation we obtain

[TABLE]

and so by the second condition $S_{jj}$ is determined uniquely by $S_{rr}$. It follows that the restriction of the linear map

[TABLE]

to $\mathrm{Aut}_{\mathrm{conv}}(A)$ is injective. Therefore it is sufficient to show that $S_{rr}$ is a convex combination of the permutation matrices $P_{rr}$ obtained by restricting the automorphisms $P\in\mathrm{Aut}(A)$ to $I_{r}\times I_{r}$. By taking $i=r,j=r$ in (8) we see that $S_{rr}$ is a convex automorphism of the subgraph $A_{rr}$, and the group

[TABLE]

is a subgroup of $\mathrm{Aut}(A_{rr})$ which acts transitively on $I_{r}$. By the third assumption $A_{rr}$ has simple spectrum. Thus to show $A_{rr}$ is a convex combination of elements of $H$ it is sufficient to prove

Lemma 1**.**

If $A\in\mathcal{S}^{n}$ is a graph with simple spectrum, and $H\leq\mathrm{Aut}(A)$ acts transitively on the vertices $\mathbf{a}$, then $H=\mathrm{Aut}(A)$ and $\text{DS}(A)$ is convex exact.

We now conclude the proof of the proposition by proving the lemma. In this proof $B_{j}$ denotes the $j$-th column of the matrix $B$, and $B_{i\star}$ denotes the $i$-th row of $B$.

To prove the lemma it is sufficient to show that

[TABLE]

because this implies that

[TABLE]

which proves that $\text{DS}(A)$ is convex exact. Additionally all automorphisms $P\in\mathrm{Aut}(A)$ are in $\operatorname{conv}H$, and since permutations are extreme points of DS this can only occur if $P\in H$, and so $H=\mathrm{Aut}(A)$.

We prove (9) using an argument from [Dym and Lipman, 2016]; If $S$ is a convex automorphism of $A$ and $v$ is an eigenvector of $A$ with eigenvalue $\lambda$, Then

[TABLE]

so either $Sv=0$ or $Sv$ is an eigenvector of $A$ with eigenvalue $\lambda$. Since $A$ has simple spectrum it follows that $Sv=\alpha v$ for some $\alpha\in\mathbb{R}$. If $S=P$ is an automorphism of $A$ then $\alpha\in\{-1,1\}$.

Let $S$ be a convex automorphism of $A$. We want to show that $S\in\operatorname{conv}H$. Since $H$ acts transitively on $\mathbf{a}$, there are permutation matrices $P(1),\ldots,P(n)\in H$ such that for any vector $w\in\mathbb{R}^{n}$

[TABLE]

Denote by $V$ the matrix whose columns are the eigenvectors of $A$. Then there is a diagonal matrix $D$ and diagonal matrices $D(1),\ldots,D(n)$ such that

[TABLE]

Note that $S$ is a convex combination of $P(1),\ldots,P(n)$ if and only if $D$ is a convex combination of $D(1),\ldots,D(n)$. If $v$ is the $j$-th eigenvector of $A$ then

[TABLE]

so in particular $v$ has no zero coordinates.

From (10) we obtain

[TABLE]

and therefore

[TABLE]

Since all entries of $V$ are non-zero the only diagonal matrix solving the equation above is the zero matrix. Thus we obtain $D$ as a convex combination of $D(k)$:

[TABLE]

∎

3.2 Genericity of the sufficient conditions

In this subsection we prove the sufficient conditions of Proposition 3.1 hold generically if $G$ is reflective and has a full orbit. The first condition was proved to hold generically for any symmetry group $G$ in Theorem 3. We choose the $r$ appearing in the last two conditions of of Proposition 3.1 such that $I_{r}$ is a full orbit, or equivalently $|I_{r}|=|G|$. We begin with some preliminaries.

Preliminaries

Recall that for a symmetry group $G$ of dimension $d$, the measure $\mu_{G}$ can be defined as the restriction to $\mathcal{V}(G)$ of the $d$-dimensional Hausdorff measure $H^{d}$ on $\mathbb{R}^{n\times n}$. We cite some basic properties of the Hausdorff measure and dimension from chapter 2 in [Falconer, 2004] which will be helpful for the proof of Lemma 3.

If $C$ has Hausdorff dimension $k$ and $s>k$, then $H^{s}(C)=0$. 2. 2.

If $M\subseteq\mathbb{R}^{n}$ is a submanifold of dimension $d$, then its Hausdorff dimension is $d$ as well. 3. 3.

If $B=\cup_{i\in I}B_{i}$ and $I$ is countable, then $\dim B=\sup_{i\in I}\dim B_{i}$. 4. 4.

If $B\subseteq C$ then $\dim(B)\leq\dim(C)$. 5. 5.

If $B\subseteq\mathbb{R}^{m}$ and $f:B\to\mathbb{R}^{n}$ is Lipschitz, then $\dim f(B)\leq\dim(B)$.

An immediate consequence is that the latter inequality holds if $f$ is a $C^{1}$ function defined on all of $\mathbb{R}^{m}$. To see this denote $B_{k}=B\cap\{x|\quad\|x\|\leq k\}$ and note that the restriction of $f$ to $B_{k}$ is Lipschitz. Therefore

[TABLE]

For Lemma 4 we will need the following simple lemma. We include a proof for completeness:

Lemma 2**.**

If $p(x)$ is a non-zero multivariate polynomial $p:\mathbb{R}^{d}\to\mathbb{R}$, then the set $\{x|\;p(x)=0\}$ has Lebesgue measure zero.

Proof.

By induction. For $d=1$ the claim is obvious. We assume the claim holds for $d-1$ and show it holds for $d$. Rewrite $p$ as

[TABLE]

By the induction hypothesis the set

[TABLE]

has measure zero in $\mathbb{R}^{d-1}$. For any fixed $(x_{1},\ldots,x_{d-1})$ in the complement of $C$, $p(x)$ is a univariate non-zero polynomial and has zeros in a (finite) subset of $\mathbb{R}$ of measure zero. Using Fubini’s theorem this implies that the set $\{x|\;p(x)=0\}$ has Lebesgue measure zero. ∎

Proof of genericity

A graph $A$ is in $\mathcal{V}(G)$ if it is symmetric and (8) is satisfied when $S$ is replaced with all permutations $P\in G$. This means that $\mathcal{V}(G)=\oplus_{i\geq j}\mathcal{V}_{ij}(G)$ where

[TABLE]

and for $i>j$

[TABLE]

Thus to prove the second condition is generic it is sufficient to show that almost every $A_{rr}\in\mathcal{V}_{rr}(G)$ has simple spectrum, and that to prove the third condition is generic we need to show that almost every $A_{rj}\in\mathcal{V}_{rj}(G)$ has full rank. Thus the second condition follows by setting $A=A_{rj}$ and $\mathcal{V}=\mathcal{V}_{rj}$ in the following Lemma:

Lemma 3**.**

If $G$ is reflective then almost all $A\in\mathcal{V}(G)$ has simple spectrum.

Proof.

Since $G$ is reflective all $P\in G$ satisfy

[TABLE]

Members $P,Q\in G$ commute because

[TABLE]

Thus $G$ can be diagonalized simultaneously, and so we can partition $\mathbb{R}^{n}$ into a direct sum of eigenspaces $\mathbb{R}^{n}=\oplus_{i=1}^{\ell}W_{i}$. We denote the dimension of each eigenspace by $d_{i}$. Select for each subspace $W_{i}$ a matrix $V_{i}\in\mathbb{R}^{n\times d_{i}}$ whose columns form an orthogonal eigenbasis of $W_{i}$, and denote

[TABLE]

A graph $A$ is in $\mathcal{V}(G)$ if and only if it is symmetric and it commutes with the members of $G$. This in turn occurs if and only if $A$ and all members of $G$ can be diagonalized simultaneously, and so there are symmetric matrices $\bar{A}_{i},\quad i=1,\ldots,\ell$ such that

[TABLE]

It follows that $\mathcal{V}(G)$ can be identified with $\oplus_{i=1}^{\ell}S(d_{i})$, and is thus of dimension

[TABLE]

For $(U_{i},\lambda_{i})\in\mathcal{O}(d_{i})\times\mathbb{R}^{d_{i}}$ we define $D(\lambda_{i})$ to be the diagonal matrix whose diagonal entries are $\lambda_{i}$, and define

[TABLE]

Now consider the function $f:\prod_{i=1}^{\ell}\left(\mathcal{O}(d_{i})\times\mathbb{R}^{d_{i}}\right)\to\mathbb{R}^{n\times n}$ defined by

[TABLE]

The image of $f$ is precisely $\mathcal{V}(G)$. Moreover the dimension of the domain of $f$ is

[TABLE]

The complement of the set of graphs $A\in\mathcal{V}(G)$ with simple spectrum is a union of sets of the form $f(E_{qr})$ where

[TABLE]

Since the dimension of each such set is strictly smaller than the dimension of the domain we obtain:

[TABLE]

and so the complement of the set of graphs $A\in\mathcal{V}(G)$ with simple spectrum is dimension deficient and thus has zero Hausdorff measure. ∎

We now prove the third condition holds generically.

Lemma 4**.**

If $G$ has a full orbit $I_{r}$ then almost every $A_{rj}\in\mathcal{V}_{rj}(G)$ has full rank.

Proof.

In this proof we denote members of the vector space $\mathcal{V}_{rj}(G)$ by $\bar{A}$. We identify this vector space with $\mathbb{R}^{\ell}$ for some $\ell$ via a linear isomorphism $\bar{A}:\mathbb{R}^{\ell}\to\mathcal{V}_{rj}(G)$, and define a multivariate polynomial $p:\mathbb{R}^{\ell}\to\mathbb{R}$ by

[TABLE]

Note that $p(x)=0$ if and only if $\bar{A}(x)$ has full rank. Thus due to Lemma 2 it is sufficient to show that $p$ isn’t identically zero, or equivalently, establish the existence of a full rank matrix in $\mathcal{V}_{rj}(G)$. We now construct such a matrix which we will denote by $\hat{A}$.

Note that $\hat{A}\in\mathcal{V}_{rj}$ if and only if

[TABLE]

This means that the values of $\hat{A}$ are required to be constant along the orbits of the action of the group $G$ on $I_{r}\times I_{j}$ defined by

[TABLE]

We choose some orbit $[(s,q)]$ of this action and define $\hat{A}$ by the requirement that $\hat{A}_{ij}=1$ if $(i,j)$ are member of this orbit, and otherwise $\hat{A}_{ij}=0$. By construction $\hat{A}\in\mathcal{V}_{rj}(G)$ and it remains to verify that it has full rank. The orbit $[(s,q)]$ has $n_{r}=|G|$ elements $(s_{1},q_{1}),\ldots,(s_{n_{r}},q_{n_{r}})$ where the $s_{i}$ are all distinct, and we can order the orbit so that the first $n_{j}$ elements of the sequence $q_{i}$ are distinct as well. Thus

[TABLE]

and so $\operatorname{rank}\hat{A}=n_{j}$. ∎

4 Almost all or nothing

In this section we prove Theorem 2; we show that for any symmetry group $G$, either the DS relaxation is never convex exact for any $A\in{\mathcal{A}}(G)$, or the DS relaxation is convex exact for almost every $A\in{\mathcal{A}}(G)$. We then explain how generic convex exactness can be established/refuted for a given group $G$.

Our proof uses another notion of exactness which we will call affine exactness: The affine automorphisms of a graph $A$ are the members of the affine set

[TABLE]

We note that affine automorphisms and convex automorphisms differ in two aspects: On the one hand the entries of convex automorphisms are required to be non-negative while the entries of affine automorphisms are not. On the other hand, affine automorphisms must be members of $\mathcal{N}(\mathrm{Aut}(A))$, a requirement we do not impose on convex automorphisms (although by Theorem 3 convex automorphisms will ”usually” satisfy this property).

We say that affine exactness holds at $A$ if

[TABLE]

We begin by establishing a connection between affine exactness and convex exactness:

Proposition 4.1**.**

For any graph $A\in{\mathcal{A}}(G)$, convex exactness holds at $A$ if and only if affine exactness holds at $A$ and

[TABLE]

Note that the RHS of (13) is always contained in the LHS.

Proof.

If affine exactness and (13) hold then

[TABLE]

so convex exactness holds as well.

Now assume convex exactness holds. Then (13) holds since

[TABLE]

To show affine exactness holds we choose some $S\in\mathrm{Aut}_{\mathrm{aff}}(A)$ and show that $S\in\mathrm{aff}\mathrm{Aut}(A)$. The ”centroid” convex automorphism $S_{c}$ is non-zero in all coordinates except for coordinates $(i,j)$ which satisfy $P_{ij}=0$ for all automorphisms $P$. Since at such coordinates $S$ is also zero, there is some $\epsilon>0$ such that

[TABLE]

is doubly stochastic. $S_{1}$ is also an affine automorphism, and thus is a convex automorphism. By assumption $S_{1}$ is a convex combination of members of $\mathrm{Aut}(A)$. In particular $S_{1},S_{c}\in\mathrm{aff}\mathrm{Aut}(A)$ and therefore so is

[TABLE]

∎

Note that the condition (13) depends only on $G$ and not on a specific choice of $A\in{\mathcal{A}}(G)$. Thus if the condition does not hold then $\text{DS}(A)$ will not be convex exact for any $A\in{\mathcal{A}}(G)$. If $G$ is such that the condition does hold then convex exactness for specific $A\in{\mathcal{A}}(G)$ is equivalent to affine exactness. Thus Theorem 2 follows from the following proposition:

Proposition 4.2**.**

Let $G$ be a symmetry group. Then either affine recovery holds for almost every $A\in{\mathcal{A}}(G)$, Or affine recovery fails for all $A\in{\mathcal{A}}(G)$.

Proof of Proposition 4.2.

As in the proof of Lemma 4 we show the set of $A\in{\mathcal{A}}(G)$ for which affine recovery fails is a null set of a suitable multivariate polynomial.

For given $A\in{\mathcal{A}}(G)$, the affine automorphisms of $A$ are the matrices $S$ satisfying the affine equations defining $\mathrm{Aut}_{\mathrm{aff}}(A)$. Denoting the map which identifies $n\times n$ matrices $S$ with $n^{2}\times 1$ vectors by $\mathrm{vec}(\cdot)$, these equations can be written in the form

[TABLE]

where $F(A)$ depends linearly on $A$. Since $A\in{\mathcal{A}}(G)$, all members of $\mathrm{vec}(G)$ are solutions of (14). Thus the kernel of $F(A)$ always includes

[TABLE]

and affine exactness holds iff $\mathrm{Ker}(F(A))=W$. Let

[TABLE]

be a unitary matrix, such that $U_{0}$ forms an orthonormal basis of $W$. Then affine exactness holds iff $F(A)U_{1}$ has full rank. Now pick some linear isometry $x\mapsto\bar{A}(x)$ from $\mathbb{R}^{\ell}$ to the vector space $\mathcal{V}(G)$. affine exactness holds at $\bar{A}(x)$ iff $x$ is not a zero of the multivariate polynomial

[TABLE]

This concludes the proof of the proposition, due to Lemma 2. ∎

Checking exactness for given groups

We now explain how we check whether convex exactness holds generically for the groups $G_{i},i=1,\ldots,9$ defined by the shapes in Figure 2. We first note that condition (13) holds for these groups. This is because $G_{i}$ all contain a full orbit. If $G$ has a full orbit $[\mathbf{a}_{k}]$ and $S$ is an affine combination of members of $G$, then the coefficients of the affine combination are just the values of the column $S_{k}$. In particular if $S$ is doubly stochastic then the affine combination is in fact a convex combination since $S_{k}$ is a probability vector.

Since (13) holds we have generic convex exactness for $G_{i}$ if and only if the polynomial $p=p(G_{i})$ is a non-zero polynomial. This can be checked either by computing the polynomial symbolically or by evaluating it on random input. We used the latter method: For each of the groups $G_{i}$ we generated $100$ random graphs in $\mathcal{V}(G_{i})$ and evaluated the polynomial on these graphs. For groups $G_{1},\ldots,G_{3}$ all $100$ evaluations of the polynomial were zero, and for the remaining graphs the polynomial was found to be non-zero at all $100$ evaluated points. These results are summarized in the first column of Table 1.

5 Centroid

Recall that the centroid solution $S_{c}$ is the matrix obtained by averaging over all members of $G$ as defined in 6. In this section we show that for any symmetry group $G$ and almost every ${\mathcal{A}}\in{\mathcal{A}}(G)$ the centroid solution $S_{c}$ can be recovered efficiently. We also show that $S_{c}$ is the solution which will be obtained by interior-point methods when solving $\text{DS}(A)$.

We begin by giving an explicit construction of $S_{c}$. Let us denote as before the equivalence classes of the action of $G$ on $\mathbf{a}$ by $I_{1},\ldots,I_{k}$. Assume that the vertices are arranged so that

[TABLE]

Set $n_{j}=|I_{j}|,j=1,\ldots,k$, and for any integer $p$ let $J_{p}\in\mathbb{R}^{p\times p}$ be the constant matrix whose entries are all $p^{-1}$. Note that $S_{c}$ is in $\mathcal{N}(G)$ and is invariant under multiplication from the left or right by elements of $G$. The only doubly stochastic matrix satisfying these properties is

[TABLE]

and therefore $S_{c}=S_{0}$. Recall that for any symmetry group $G$ and almost every $A\in{\mathcal{A}}(G)$ the vector $s(A)$ is discriminative. In this case the centroid solution $S_{c}=S_{0}$ can be easily computed without even solving the DS relaxation: We first construct a matrix $S$ by setting $S_{ij}=1$ if $s_{i}(A)=s_{j}(A)$ and $S_{ij}=0$ otherwise. We can then obtain $S_{c}$ by normalizing the rows of $S$.

Interior point algorithms

Interior point algorithms solve (2) by solving problems of the form

[TABLE]

and taking $\alpha\rightarrow 0$ to obtain a solution for (2). The function $F$ is chosen so that it explodes at the boundary, and so the constraints $S\geq 0$ will never be active in (15). A common choice [Wright and Nocedal, 1999] for $F$ is $F(S)=-\sum_{ij}\log(S_{ij})$. Specialized solvers for (2) such as [Rangarajan et al., 1996, Solomon et al., 2016] often use $F(S)=\sum_{ij}S_{ij}\log S_{ij}$. Note that while this $F$ does not explode at the boundary, its derivatives do.

To include both choices of $F$, is well as other possible choices, we will deal with general $F$ which are of the form

[TABLE]

where $f:\mathbb{R}_{\geq 0}\to\mathbb{R}\cup\{\infty\}$ is continuous and strictly convex and $f(t)<\infty$ if $t>0$.

Theorem 4**.**

Let $G$ be a symmetry group, and $F$ be a function satisfying the conditions described previously. Then for almost every $A\in{\mathcal{A}}(G)$ the unique minimizers $S^{*}_{\alpha}$ of (15) converge to $S_{c}$ as $\alpha$ tends to zero.

Proof.

We assume that $\text{DS}(A)$ is weakly exact. This assumption holds for almost every $A\in{\mathcal{A}}(G)$. By passing to a subsequence we can assume that $S_{\alpha}^{*}$ converges to some $S^{*}$ in the compact set DS. We need to show that $S^{*}=S_{c}$.

We note that for any $S\in\text{DS},P\in G,\alpha>0$

[TABLE]

Since $S_{\alpha}^{*}$ is the unique minimizer of $E_{\alpha}$ this equality implies that $S_{\alpha}^{*}$ is invariant under multiplication by elements of $G$ from the right and the left. Thus this is true for $S^{*}$ as well. Due to continuity $F$ is bounded from below and so it can be shown that

[TABLE]

It follows that $S^{*}$ is a convex automorphism and since $\text{DS}(A)$ is weakly exact $S^{*}\in\mathcal{N}(G)$. Since we also showed $S^{*}$ to be invariant under multiplication by $G$ from the left and right it follows that $S^{*}=S_{c}$.

∎

6 Retrieving isomorphisms

In this section we discuss how convex exactness can be used to retrieve isomorphisms. We will discuss two classes of methods. The first class searches for extreme points of the convex set of convex isomorphisms. We will show that under the assumptions of Proposition 3.1 all isomorphisms of the graphs can be retrieved quite efficiently. However finding extreme points is not a stable methods for retrieving isomorphisms once noise is introduced. This leads to the second class of methods, which we call projection methods. Projection methods are the methods typically used in practice to achieve a permutation solution from the original solution of the DS relaxation. We show theoretically that the popular ”convex to concave” projection method is able to retrieve a correct isomorphism, and explore experimentally the behavior of this method as well as the $L_{2}$ projection method when noise is introduced.

6.1 Finding extreme points

When convex exactness holds, finding an isomorphism is reduced to the problem of finding an extreme point of the optimal set $\mathrm{ISO}_{\mathrm{conv}}(A,B)$ defined by the linear constraints

[TABLE]

An extreme point(=basic feasible solution) of this linear feasibility program can be found using the simplex algorithm. Extreme points can also be found using interior point algorithms by optimizing a random linear energy over $\mathrm{ISO}_{\mathrm{conv}}(A,B)$. In [Dym and Lipman, 2016] a similar problem is discussed, and it is shown that if the linear energies are randomly drawn from the uniform distribution on $S^{n^{2}-1}$, then with probability one the obtained linear program will have a unique solution, which will be an extreme point. Moreover all extreme points will be obtained with equal probability.

Table 1 shows the successfulness of the latter method in returning isomorphisms for symmetric problems in which convex exactness holds. For each of the nine symmetry groups $G_{i}$ defined by the shapes in Figure 2 we generated $100$ random graphs in $\mathcal{V}(G_{i})$ according to the distribution $\mu_{G_{i}}$. For each such graph we then found an extreme point by maximizing a random linear energy over the optimal set. As shown in Table 1 for the graphs $G_{i},i>3$ for which convex exactness holds generically, this algorithm succeeded in returning a permutation in all $100$ experiments. For the groups $G_{i},i=1,2,3$ for which convex exactness does not hold, this algorithm returned permutations in more than half of the experiments, but non-integer solutions were also obtained. This is due to the fact that the optimal set contains non-integer extreme points in this case.

Next we suggest a more efficient method for obtaining all extreme points of the set of convex isomorphisms, under the assumption that the assumptions of Proposition 3.1 hold and $k=|\mathrm{ISO}(A)|$ is not too large $k<<n$.

If $s(A)$ is discriminative, then the centroid solution $S_{c}$ can be found directly as described in the previous section.

Once a convex isomorphism $S=S_{c}$ was found, we use the technique of [Pataki, 1996] to find an extreme point. We now describe this technique:

We begin with some preliminaries: For $S,T\in\mathbb{R}^{n\times n}$, we say that $S\preceq T$ if $S_{ij}=0$ whenever $T_{ij}=0$. We say that $S\prec T$ if $S\preceq T$ but the converse inequality $T\preceq S$ does not hold.

A face of a convex set $K$ is a subset $F\subseteq K$ such that for all $x,y\in K$ and $t\in(0,1)$ satisfying

[TABLE]

necessarily $x,y\in F$. An extreme point is a face which is a singleton. If $K$ is a convex compact set then it is the convex hull of its extreme points $E$. Moreover, for each face $F\subseteq K$,

[TABLE]

Every $S\in\mathrm{ISO}_{\mathrm{conv}}(A,B)$ defines a face

[TABLE]

and an affine space obtained from $F(S)$ by removing the positivity constraints, i.e.,

[TABLE]

We note that $S$ is in the relative interior of $F(S)\subseteq V(S)$. This means that for all $R\in V(S)$ there is a sufficiently small $t>0$ such that $(1-t)S+tR\in F(S)$. The boundary of $F(S)$ in $V(S)$ is the set:

[TABLE]

We can now describe the algorithm of [Pataki, 1996]:

We are given as input some $S\in\mathrm{ISO}_{\mathrm{conv}}(A,B)$ and set $r=0$ and $S_{r}=S$. 2. 2.

We compute a spanning subset to the affine space $V(S_{r})$. If $V(S_{r})=\{S_{r}\}$ then $S_{r}$ is an extreme point and we are done. 3. 3.

Otherwise we choose some $R_{r}\neq S_{r}$ in $V(S_{r})$. We then find the unique $t>0$ such that

[TABLE]

and set $S_{r+1}$ to be the matrix on the left hand side. We then return to the previous step.

The iterative process can only terminate when $V(S_{r})=\{S_{r}\}$. This will necessarily occur after a finite number of steps since $S_{r+1}$ always has more zeros than $S_{r}$. In the convex exact case, a permutation will be attained within $k=|\mathrm{ISO}(A,B)|$ steps. This is because each face $F(S_{r+1})$ is strictly contained in the former face $F(S_{r})$ and therefore according to (16) the number of extreme points=permutations in $F(S_{r+1})$ is strictly smaller than the number of extreme points in $F(S_{r})$.

Once a permutation $P(1)$ is obtained, an additional permutation can be sought for by repeating the process above, but beginning with $S_{0}^{1}=(1-t)S_{0}+tP(1)$ where $t<0$ is the smallest possible so that $S_{0}(t)$ is doubly stochastic. This choice gives an initial convex isomorphism such that $P(1)\not\in F(S_{0})$, guaranteeing that the algorithm will return a new permutation $P(2)$. In the next step we can set $S_{0}^{2}=(1-t)S_{0}^{1}+tP(2)$ and continue in this manner until we obtain a collection of isomorphisms $P(1),\ldots,P(L)$, and $S_{c}$ is a convex combination of these isomorphisms. In fact under the full orbit assumption $P(1),\ldots,P(L)$ will be all the isomorphisms. This is because $S_{c}$ can be written as a positive convex combination of all members of $\mathrm{ISO}(A,B)$, and the members of $\mathrm{ISO}(A,B)$ are linearly independent, implying that this is the only possible convex combination giving $S$, so that that all isomorphisms were obtained. The linear independence of $\mathrm{ISO}(A,B)$ follows from the fact that it has full orbit, and so each isomorphism has a non-zero coordinate $i,j$ on which all other isomorphisms vanish.

From a computational perspective, under the conditions of Proposition 3.1, The algorithm above will return an isomorphism within $k$ steps, and all isomorphisms within $O(k^{2})$ iterations. Computing the first affine space $V(S_{c})$ is basically the problem of finding a linear basis to the solution set of the linear equations defining $V(S_{c})$. Since $S_{c}$ has at most $nk$ non-zero entries, this is a linear equation in $O(n)$ variables instead of $n^{2}$ variables. For finding the subsequent affine spaces $V(S_{r})$ additional computational saving can be obtained due to the fact that $V(S_{r})\subseteq V(S_{c})$. Thus all elements in $V(S_{r})$ are affine combinations of $k+1$ spanning element of the affine space $V(S_{c})$, so $V(S_{r})$ is obtained by solving a linear equation in only $k+1$ variables.

Figure 4 shows the results of applying the algorithm described above to find the symmetries of a $20\times 25$ grid. The grid has a reflective symmetry group with full orbit and thus fulfills the conditions of Theorem 1. We took $A=B$ to be the Euclidean distance matrix of the grid (here $n=500$) and used the algorithm described above to obtain all symmetries of the grid. In our implementation in Matlab this calculation took around ten seconds.

6.2 Projection methods

The classical approach [Aflalo et al., 2015] for projecting a permutation solution from the doubly stochastic relaxation is using the standard $L_{2}$ projection, which can be implemented as a linear program and solved efficiently using the Hungarian algorithm. See [Zaslavskiy et al., 2009] for more details. A more accurate and more computationally demanding method is the ”convex to concave” method. We will explain this method in the formulation used in the DS++ algorithm [Dym et al., 2017]. Similar suggestions appear in [Zaslavskiy et al., 2009, Ogier and Beyer, 1990]. We then prove DS++ obtains a permutation solution in the convex exact case (up to some technicalities which will be explained), and examine the behavior of both projection methods when noise is added.

Convex to concave projection

The convex to concave method sequentially solves optimization problems of the form

[TABLE]

The strictly concave function

[TABLE]

is non-negative on DS, and $g(S)=0$ if and only if $S$ is a permutation. Additionally if $a$ is sufficiently large so that $E(S,a)$ is strictly concave, then the (global and local) minima of (17) will necessarily be permutations since the minima of a strictly concave function on a convex compact set are always extreme points. Thus the global minimum of the relaxed $\eqref{e:dspp}$ and the original quadratic assignment problem are identical. Note however that since (17) is not convex computing the global minimum is no longer tractable.

Building on this observation, the convex to concave method minimizes (locally) a sequence of optimization problems of the form (17) on a sequence of choices

[TABLE]

and in each step uses the obtained solution $S_{i}$ as a warm start to the optimization of $E(S,a_{i+1})$. The first point $a_{0}$ is selected so that $E(S,a_{0})$ is convex, and the last point is selected so that $E(S,a_{N})$ is strictly concave and thus the obtained local minima $S_{N}$ is guaranteed to be a permutation.

The first point $a_{0}$ can be selected to be zero to ensure that $E(S,a_{0})$ is convex. However a better selection is $a_{0}=\lambda_{\mathrm{min}}$ where $\lambda_{\mathrm{min}}\geq 0$ is the minimal eigenvalue of the quadratic form

[TABLE]

when restricted to the subspace

[TABLE]

Similarly the last point $a_{N}$ is selected to be (slightly larger than) the maximal eigenvalue $\lambda_{\mathrm{max}}$ of the same quadratic form over the same subspace. This choice ensures that $E(S,a_{N})$ is (strictly) concave. The remaining points $a_{i}$ can be uniformly sampled in the interval $[a_{0},a_{N}]$ (for lack of a better strategy).

Note that if $A$ and $B$ are isomorphic, then for any $a>0$ the global minimizers of $E(S,a)$ are precisely $\mathrm{ISO}(A,B)$ (while for $a=0$ the global minimizers are $\mathrm{ISO}_{\mathrm{conv}}(A,B)$ ). This observation suggests the ”convex to concave” method may be successful in retrieving isomorphisms even for symmetric problems, and possibly could return integer solutions $S_{i}$ even for $i<N$. We now give a theoretical justification for these observations.

We assume that we obtain each $S_{i}^{*}$ from a local minimization algorithm with the following properties:

Monotonicity: $E(S_{i}^{*},a_{i})\leq E(S_{i-1}^{*},a_{i})$. 2. 2.

The first-order necessary condition (KKT conditions) for local minima is satisfied at $S_{i}^{*}$. 3. 3.

The second-order necessary condition for local minima of $E(\cdot,a_{i})$ is satisfied at $S_{i}^{*}$. That is

[TABLE]

Here $H$ is the Hessian of the quadratic form $E(\cdot,a_{i})$.

Under these assumptions we prove

Theorem 5**.**

Assume $A$ and $B$ are isomorphic and the DS relaxation is convex exact at $A$. Assume $S_{i}^{*},i=0,\ldots,N$ satisfy conditions (1)-(3). Then if $a_{1}$ is sufficiently close to $a_{0}$

[TABLE]

The theorem is proved in Appendix A.

Isomorphism retrieval for noisy problems

We examine the behavior of the DS relaxation coupled with the projections described above for noisy symmetric problems by conducting the following experiment:

We construct a random bilaterally symmetric graphs $A\in\mathbb{R}^{n\times n}$ and choose $B=A$. We then perturb these graphs by two randomly selected symmetric matrices $\Delta A,\Delta B$, and solve the DS relaxation using both projection methods. We do this for $n=10,30,50$ and for matrices $\Delta A,\Delta B$ with Frobenius norm $\epsilon=10^{\alpha}$ where we use ten values of $\alpha$ uniformly chosen from the interval $[-3,0]$. The graph $A$ is chosen by computing an isometry $L:\mathbb{R}^{k}\to\mathcal{V}(G)$ where $G$ is a permutation subgroup with two elements, and then sampling a vector $x\in\mathbb{R}^{k}$ uniformly from the unit sphere to obtain $A=L(x)$. For each fixed value of $n,\alpha$ we repeat $100$ different instances of the experiment, and compute the retrieval ratio of both methods, which we define as the number of times the method returned a permutation from $G$ divided by the number of experiments (100). The results are shown in Figure 5.

It can be seen that both methods succeed in retrieving a correct permutation at low noise levels, but the convex-to-concave method (denoted by DS++) is more successful than the $L_{2}$ projection method (denoted by DS) at higher noise levels. In the case $n=10$ we also add the ”groud truth retrieval ratio”, that is the number of instances in which the global minimizer of the graph matching energy was indeed in $G$ divided by the number of experiments. It can be seen that as the noise level approaches $10^{0}=1$ the noise ”takes over the problem” and the members of $G$ are no longer the global minimizers. The ground truth solutions was obtained by the semi-definite relaxation of [Kezurer et al., 2015] which is known to be very tight, though computationally expensive. We verify that the solution obtained from the semi-definite relaxation is indeed the correct solution by checking that the difference between the lower bound provided by the relaxation and the upper bound provided by projecting the solution of the relaxation are negligible.

As a side note, we observe that at low noise levels DS++ obtains a solution in $G$ after two iterations in accordance with Theorem 5, and that even at high noise levels a permutation solution is usually attained after four iterations. This indicates that it might be worthwhile to choose a smaller $a_{N}$, or alternatively to consider less steps in the convex-to-concave process.

Appendix A Convex to concave

Proof of Theorem 5.

We begin with some preliminaries. First note that if $S_{i-1}^{*}$ is an isomorphism, then $E(S_{i}^{*},a_{i})=0$ due to the monotonicity condition and thus $S_{i}^{*}$ is an isomorphism.

In the asymmetric case the claim is trivial: Since $E(S,a_{0})$ is convex its local minimizers are also global minimizers. Since in the asymmetric case is the unique isomorphism between $A$ and $B$ is the only global minimizer for any $a_{0}\geq 0$ it follows that $S_{0}^{*}$ is that unique minimizer. Therefore in this proof we will focus on the symmetric case only.

In the symmetric case there are at least two isomorphisms $P_{0},P_{1}$. Thus $P_{1}-P_{0}$ is an eigenvector of the energy $E(S)$ with eigenvalue $\lambda_{\mathrm{min}}=0$ and so $a_{0}=0$.

Our claim follows easily from the following lemma:

Lemma 5**.**

There exists an open set $U$ containing $\mathrm{ISO}_{\mathrm{conv}}(A,B)$ such that for all $a>0$, The only points satisfying the first and second order conditions for local minimization of $E(S,a)$ are the members of $\mathrm{ISO}(A,B)$.

To obtain the theorem from the lemma, let $m>0$ be the minimum of $E(S)$ on the compact set $DS\setminus U$. For any $a_{1}>0$ sufficiently small so that $a_{1}\max_{S\in DS}g(S)<m$ we obtain

[TABLE]

It follows that $S_{1}^{*}\in U$ and since it satisfies the first and second order conditions for local minimization of $E(\cdot,a_{1})$ it follows that $S_{1}^{*}\in\mathrm{ISO}(A,B)$.

Proof of Lemma 5.

We construct for each $S\in\mathrm{ISO}_{\mathrm{conv}}(A,B)$ an open set $U_{S}$ satisfying the properties required from $U$ and then choose

[TABLE]

If $S$ is a convex isomorphism but not a permutation we choose

[TABLE]

Fix some $Q\in U_{S}$, we claim that the second-order necessary condition for minimizing $E(\cdot,a)$ is not satisfied at $Q$ for any $a>0$. Since $S$ is a convex combination of isomorphisms we can choose an isomorphism $P$ such that $S\succeq P$, and so $P,S\in F(Q)$. Since $P,S$ are both zeros of the convex quadratic form $E(\cdot,0)$, it follows that the second-order condition does not hold since (denoting by $H_{g}$ the Hessian of $g$ )

[TABLE]

For isomorphisms $P$ we choose $U_{P}$ as follows:

For any $S\in\text{DS}\setminus\Pi_{n}$ the concavity of $g$ implies

[TABLE]

In particular this is true for any $S$ in the compact set

[TABLE]

Since $g$ is $C^{1}$ the function

[TABLE]

is continuous. Thus, there is a neighborhood $U_{P}\subseteq\{S\in DS|\sup_{i,j}|P_{ij}-S_{ij}|\leq\frac{1}{4}\}$ of $P$ on which

[TABLE]

Fix some $Q\in U_{P}$. Define

[TABLE]

Note that $\sup_{i,j}\left|P_{ij}-Q_{ij}(t_{M})\right|=1$ and therefore there is some $1<t_{0}<t_{M}$ such that $S=Q(t_{0})\in K$. It follows from (19) that

[TABLE]

and therefore

[TABLE]

The convexity of $E$ implies that for all $Q\in U_{P}$,

[TABLE]

From the last two equations it follows that for any $a>0$ the energy $E(\cdot,a)$ has a descent direction $P-Q$ at any point $Q\in U_{P}\setminus\{P\}$. Not that this direction is orthogonal to the gradients of the constraints defining DS, since $Q+t(P-Q)$ is feasible if $|t|$ is small enough. Thus the first-order condition does not hold at $Q$. ∎