Product Graph-based Higher Order Contextual Similarities for Inexact   Subgraph Matching

Anjan Dutta; Josep Llad\'os; Horst Bunke; Umapada Pal

arXiv:1702.00391·cs.CV·February 2, 2017

Product Graph-based Higher Order Contextual Similarities for Inexact Subgraph Matching

Anjan Dutta, Josep Llad\'os, Horst Bunke, Umapada Pal

PDF

TL;DR

This paper introduces a novel approach to inexact subgraph matching by leveraging higher-order contextual similarities computed via tensor product graphs, resulting in more reliable and discriminative matching solutions.

Contribution

It proposes a new method using tensor product graphs to incorporate higher-order contextual information into subgraph matching, improving accuracy over traditional pairwise methods.

Findings

01

Enhanced discriminative power of node and edge similarities.

02

Improved subgraph matching accuracy on synthetic and real benchmarks.

03

Higher order contextual similarities lead to better approximate solutions.

Abstract

Many algorithms formulate graph matching as an optimization of an objective function of pairwise quantification of nodes and edges of two graphs to be matched. Pairwise measurements usually consider local attributes but disregard contextual information involved in graph structures. We address this issue by proposing contextual similarities between pairs of nodes. This is done by considering the tensor product graph (TPG) of two graphs to be matched, where each node is an ordered pair of nodes of the operand graphs. Contextual similarities between a pair of nodes are computed by accumulating weighted walks (normalized pairwise similarities) terminating at the corresponding paired node in TPG. Once the contextual similarities are obtained, we formulate subgraph matching as a node and edge selection problem in TPG. We use contextual similarities to construct an objective function and…

Tables1

Table 1. Table 1: Comparative results with different existing methods

Methods	P	R	F	AveP	T
SSGR [64]	41.00	80.00	54.21	64.45	-
ILPIso [21]	65.44	58.23	59.11	57.75	27.29s
FGE [65]	56.00	100.00	71.79	75.50	-
SG [8]	54.19	83.98	65.87	65.43	0.07s
NCRAG [66]	61.89	82.87	70.85	70.65	0.72s
PG-N	61.60	72.81	64.94	60.21	53.60
PG-R	69.90	84.95	78.78	81.10	33.43
PG-B	70.56	86.29	80.10	82.98	33.37s

Equations63

K (i_{1} i_{2}, j_{1} j_{2}) = ⎩ ⎨ ⎧ κ_{i_{1} i_{2}}^{v}, κ_{i_{1} i_{2}, j_{1} j_{2}}^{e}, 0, if i_{1} = j_{1} and i_{2} = j_{2} . if (u_{i_{1}}, v_{j_{1}}) \in E_{1} and (u_{i_{2}}, v_{j_{2}}) \in E_{2} . otherwise .

K (i_{1} i_{2}, j_{1} j_{2}) = ⎩ ⎨ ⎧ κ_{i_{1} i_{2}}^{v}, κ_{i_{1} i_{2}, j_{1} j_{2}}^{e}, 0, if i_{1} = j_{1} and i_{2} = j_{2} . if (u_{i_{1}}, v_{j_{1}}) \in E_{1} and (u_{i_{2}}, v_{j_{2}}) \in E_{2} . otherwise .

X \in Π max J_{g m} (X) = vec (X)^{T} K vec (X)

X \in Π max J_{g m} (X) = vec (X)^{T} K vec (X)

Π = {X \in {0, 1}^{n_{1} \times n_{2}} : X 1_{n_{2}} = 1_{n_{1}}, X^{T} 1_{n_{1}} = 1_{n_{2}}}

Π = {X \in {0, 1}^{n_{1} \times n_{2}} : X 1_{n_{2}} = 1_{n_{1}}, X^{T} 1_{n_{1}} = 1_{n_{2}}}

D = {X \in R^{n_{1} \times n_{2}} : X \geq 0, X 1_{n_{2}} = 1_{n_{1}}, X^{T} 1_{n_{1}} = 1_{n_{2}}}

D = {X \in R^{n_{1} \times n_{2}} : X \geq 0, X 1_{n_{2}} = 1_{n_{1}}, X^{T} 1_{n_{1}} = 1_{n_{2}}}

V_{X} = {(u_{i}, u_{j}) : u_{i} \in V_{1}, u_{j} \in V_{2}}

V_{X} = {(u_{i}, u_{j}) : u_{i} \in V_{1}, u_{j} \in V_{2}}

E_{X} = {((u_{i}, u_{j}), (v_{i}, v_{j})) : (u_{i}, v_{i}) \in E_{1} and (u_{j}, v_{j}) \in E_{2}, u_{i}, v_{i} \in V_{1}, u_{j}, v_{j} \in V_{2}}

E_{X} = {((u_{i}, u_{j}), (v_{i}, v_{j})) : (u_{i}, v_{i}) \in E_{1} and (u_{j}, v_{j}) \in E_{2}, u_{i}, v_{i} \in V_{1}, u_{j}, v_{j} \in V_{2}}

S_{p_{X}} = p_{X} ⊙ (n \to \infty lim k = 0 \sum n W_{X}^{k}) 1

S_{p_{X}} = p_{X} ⊙ (n \to \infty lim k = 0 \sum n W_{X}^{k}) 1

S W_{X}^{C S} = p_{X} ⊙ (n \to \infty lim k = 0 \sum n λ^{k} W_{X_{k}}) 1 = p_{X} ⊙ (I - λ W_{X})^{- 1} 1

S W_{X}^{C S} = p_{X} ⊙ (n \to \infty lim k = 0 \sum n λ^{k} W_{X_{k}}) 1 = p_{X} ⊙ (I - λ W_{X})^{- 1} 1

W_{X_{k}} = W_{X}^{k}

W_{X_{k}} = W_{X}^{k}

W = 0 w^{12} w^{13} w^{21} 0 w^{23} w^{31} w^{32} 0

W = 0 w^{12} w^{13} w^{21} 0 w^{23} w^{31} w^{32} 0

p = p^{1} p^{2} p^{3}

p = p^{1} p^{2} p^{3}

p ⊙ (I + W_{1} + W_{2}) 1 = p^{1} (1 + w^{12} w^{21} + w^{13} w^{31}) + p^{1} (w^{21} + w^{23} w^{31}) + p^{1} (w^{31} + w^{32} w^{21}) p^{2} (w^{12} + w^{13} w^{32}) + p^{2} (1 + w^{21} w^{12} + w^{23} w^{32}) + p^{2} (w^{32} + w^{31} w^{12}) p^{3} (w^{13} + w^{12} w^{23}) + p^{3} (w^{23} + w^{21} w^{13}) + p^{3} (1 + w^{31} w^{13} + w^{32} w^{23})

p ⊙ (I + W_{1} + W_{2}) 1 = p^{1} (1 + w^{12} w^{21} + w^{13} w^{31}) + p^{1} (w^{21} + w^{23} w^{31}) + p^{1} (w^{31} + w^{32} w^{21}) p^{2} (w^{12} + w^{13} w^{32}) + p^{2} (1 + w^{21} w^{12} + w^{23} w^{32}) + p^{2} (w^{32} + w^{31} w^{12}) p^{3} (w^{13} + w^{12} w^{23}) + p^{3} (w^{23} + w^{21} w^{13}) + p^{3} (1 + w^{31} w^{13} + w^{32} w^{23})

S W_{X}^{C S}

S W_{X}^{C S}

= p_{X} ⊙ (1 - λ^{2}) (I - λ W_{X} + λ^{2} Q_{X})^{- 1} 1

W_{X_{k}} = ⎩ ⎨ ⎧ W_{X} W_{X}^{2} - (Q_{X} + I) W_{X_{k - 1}} W_{X} - W_{X_{k - 2}} Q_{X} if k = 1 if k = 2 if k \geq 3

W_{X_{k}} = ⎩ ⎨ ⎧ W_{X} W_{X}^{2} - (Q_{X} + I) W_{X_{k - 1}} W_{X} - W_{X_{k - 2}} Q_{X} if k = 1 if k = 2 if k \geq 3

Q_{X} = w^{12} w^{21} + w^{13} w^{31} - 1 00 0 w^{12} w^{21} + w^{23} w^{32} - 1 0 00 w^{13} w^{31} + w^{23} w^{32} - 1

Q_{X} = w^{12} w^{21} + w^{13} w^{31} - 1 00 0 w^{12} w^{21} + w^{23} w^{32} - 1 0 00 w^{13} w^{31} + w^{23} w^{32} - 1

p_{X} ⊙ (I + W_{1} + W_{2}) 1 = p^{1} + p^{1} (w^{21} + w^{23} w^{31}) + p^{1} (w^{31} + w^{32} w^{21}) p^{2} (w^{12} + w^{13} w^{32}) + p^{2} + p^{2} (w^{32} + w^{31} w^{12}) p^{3} (w^{13} + w^{12} w^{23}) + p^{3} (w^{23} + w^{21} w^{13}) + p^{3}

p_{X} ⊙ (I + W_{1} + W_{2}) 1 = p^{1} + p^{1} (w^{21} + w^{23} w^{31}) + p^{1} (w^{31} + w^{32} w^{21}) p^{2} (w^{12} + w^{13} w^{32}) + p^{2} + p^{2} (w^{32} + w^{31} w^{12}) p^{3} (w^{13} + w^{12} w^{23}) + p^{3} (w^{23} + w^{21} w^{13}) + p^{3}

S_{V} (u_{i}, u_{j}) = W_{X}^{C S} (u_{i}, u_{j}), (u_{i}, u_{j}) \in V_{X}

S_{V} (u_{i}, u_{j}) = W_{X}^{C S} (u_{i}, u_{j}), (u_{i}, u_{j}) \in V_{X}

S_{E} ((u_{i}, u_{j}), (v_{i}, v_{j})) = \frac{W _{X}^{C S} ( u _{i} , u _{j} )}{Δ ^{+} ( u _{i} , u _{j} )} + \frac{W _{X}^{C S} ( v _{i} , v _{j} )}{Δ ^{+} ( v _{i} , v _{j} )}, (u_{i}, u_{j}), (v_{i}, v_{j}) \in V_{X}

S_{E} ((u_{i}, u_{j}), (v_{i}, v_{j})) = \frac{W _{X}^{C S} ( u _{i} , u _{j} )}{Δ ^{+} ( u _{i} , u _{j} )} + \frac{W _{X}^{C S} ( v _{i} , v _{j} )}{Δ ^{+} ( v _{i} , v _{j} )}, (u_{i}, u_{j}), (v_{i}, v_{j}) \in V_{X}

f (x, y) = S_{V}^{'} x + S_{E}^{'} y

f (x, y) = S_{V}^{'} x + S_{E}^{'} y

K = [S_{V} 0 0 S_{E}]

K = [S_{V} 0 0 S_{E}]

u_{2} \in V_{2} \sum x_{u_{i}, u_{j}} <= 1, \forall u_{i} \in V_{1}

u_{2} \in V_{2} \sum x_{u_{i}, u_{j}} <= 1, \forall u_{i} \in V_{1}

(u_{j}, v_{j}) \in E_{2} \sum y_{u_{i} u_{j}, v_{i} v_{j}} <= 1, \forall (u_{i}, v_{i}) \in E_{1}

(u_{j}, v_{j}) \in E_{2} \sum y_{u_{i} u_{j}, v_{i} v_{j}} <= 1, \forall (u_{i}, v_{i}) \in E_{1}

u_{i} \in V_{1} \sum x_{u_{i}, u_{j}} <= 1, \forall u_{j} \in V_{2}

u_{i} \in V_{1} \sum x_{u_{i}, u_{j}} <= 1, \forall u_{j} \in V_{2}

(u_{i}, v_{i}) \in E_{1} \sum y_{u_{i} u_{j}, v_{i} v_{j}} <= 1, \forall (u_{j}, v_{j}) \in E_{2}

(u_{i}, v_{i}) \in E_{1} \sum y_{u_{i} u_{j}, v_{i} v_{j}} <= 1, \forall (u_{j}, v_{j}) \in E_{2}

(v_{i}, v_{j}) \in V_{X} \sum y_{u_{i} u_{j}, v_{i} v_{j}} <= x_{u_{i}, u_{j}} . min (Δ^{+} (u_{i}), Δ^{+} (u_{j})), \forall (u_{i}, u_{j}) \in V_{X}

(v_{i}, v_{j}) \in V_{X} \sum y_{u_{i} u_{j}, v_{i} v_{j}} <= x_{u_{i}, u_{j}} . min (Δ^{+} (u_{i}), Δ^{+} (u_{j})), \forall (u_{i}, u_{j}) \in V_{X}

(u_{i}, u_{j}) \in V_{X} \sum y_{u_{i} u_{j}, v_{i} v_{j}} <= x_{v_{i}, v_{i}} . min (Δ^{-} (v_{i}), Δ^{-} (v_{j})), \forall (v_{i}, v_{j}) \in V_{X}

(u_{i}, u_{j}) \in V_{X} \sum y_{u_{i} u_{j}, v_{i} v_{j}} <= x_{v_{i}, v_{i}} . min (Δ^{-} (v_{i}), Δ^{-} (v_{j})), \forall (v_{i}, v_{j}) \in V_{X}

x_{u_{i}, u_{j}} \in [0, 1], \forall (u_{i}, u_{j}) \in V_{X}

x_{u_{i}, u_{j}} \in [0, 1], \forall (u_{i}, u_{j}) \in V_{X}

y_{u_{i} u_{j}, v_{i} v_{j}} \in [0, 1], \forall ((u_{i}, u_{j}), (v_{i}, v_{j})) \in E_{X}

y_{u_{i} u_{j}, v_{i} v_{j}} \in [0, 1], \forall ((u_{i}, u_{j}), (v_{i}, v_{j})) \in E_{X}

a cc = \frac{t r ( X _{a l g t hm}^{T} X _{t r u t h} )}{t r ( 1 _{∣ V_{2} ∣ \times ∣ V_{1} ∣} X _{t r u t h} )}

a cc = \frac{t r ( X _{a l g t hm}^{T} X _{t r u t h} )}{t r ( 1 _{∣ V_{2} ∣ \times ∣ V_{1} ∣} X _{t r u t h} )}

o bj = J_{g m} (X_{a l g t hm})

Peer Reviews

No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.

Videos

No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.

Full text

Product Graph-based Higher Order Contextual Similarities for Inexact Subgraph Matching

Anjan Dutta

Josep Lladós

Horst Bunke

Umapada Pal

Computer Vision Center, Universitat Autònoma de Barcelona, Edifici O, Campus UAB, 08193 Bellaterra, Barcelona, Spain

Institute of Computer Science, University of Bern, Neubrückstrasse 10, CH-3012 Bern, Switzerland

Computer Vision and Pattern Recognition Unit, Indian Statistical Institute, 203, B.T.Road, Kolkata-108, India

Abstract

Many algorithms formulate graph matching as an optimization of an objective function of pairwise quantification of nodes and edges of two graphs to be matched. Pairwise measurements usually consider local attributes but disregard contextual information involved in graph structures. We address this issue by proposing contextual similarities between pairs of nodes. This is done by considering the tensor product graph (TPG) of two graphs to be matched, where each node is an ordered pair of nodes of the operand graphs. Contextual similarities between a pair of nodes are computed by accumulating weighted walks (normalized pairwise similarities) terminating at the corresponding paired node in TPG. Once the contextual similarities are obtained, we formulate subgraph matching as a node and edge selection problem in TPG. We use contextual similarities to construct an objective function and optimize it with a linear programming approach. Since random walk formulation through TPG takes into account higher order information, it is not a surprise that we obtain more reliable similarities and better discrimination among the nodes and edges. Experimental results shown on synthetic as well as real benchmarks illustrate that higher order contextual similarities add discriminating power and allow one to find approximate solutions to the subgraph matching problem.

keywords:

Subgraph matching, Product graph, Random walks, Backtrackless walks, Contextual similarities, Graphic recognition.

1 Introduction

The use of representational models allowing to describe visual objects in terms of their components and their relations has gained interest among the research community. Relations between parts express a notion of context, so objects can be recognized even if they appear in highly distorted or noisy visual conditions. At the heart of structural models, graphs have been used over decades as robust and theoretically solid representations. When objects are represented structurally by attributed graphs, their comparison for recognition and retrieval purposes is performed using some form of graph matching. Basically, an attributed graph is a 4-tuple $G(V,E,\alpha,\beta)$ comprising a set $V$ of vertices or nodes together with a set $E\subseteq V\times V$ of edges and two mappings $\alpha:V\rightarrow\mathbb{R}^{m}$ and $\beta:E\rightarrow\mathbb{R}^{n}$ which respectively assign attributes to the nodes and edges. Given two attributed graphs, graph matching can roughly be defined as a process of finding a mapping between the node and edge sets of two graphs that satisfies some constraints. Graph matching is a very fundamental problem in theoretical computer science and it has been successfully applied to many applications in computer vision and pattern recognition [1, 2, 3, 4, 5, 6, 7, 8]. In fact, graphs are used in several domains including chemical compound, social network, biological and protein structure, program flow etc. Any complex data structure can efficiently be represented with a graph in terms of relations among different entities. Hence the theory of graph matching can be applied to different problems in any of the previously mentioned fields.

Graph matching is a very well known but challenging task. It requires high time complexity to be solved in an exact way. A particular class of algorithms resulting from the consideration of outliers111By outliers, we mean the set of nodes and edges in the target graph that are not related to nodes and edges of the pattern graph. is called subgraph matching. Roughly, it can be defined as matching one graph (pattern graph) as part of the other (target graph). Addressing this problem is justified since in computer vision and pattern recognition applications, due to the existence of background, occlusions and geometric transformations of objects, outliers do exist in many scenarios. Most of the modern algorithms formulate (sub)graph matching as an optimization of an objective function of pairwise quantification of the nodes and edges. But pairwise measurements usually consider local attributes and do not encode context information i.e. neighbourhood information of nodes and edges. As a result, pairwise quantifications are less reliable, especially in case of subgraph matching scenarios [9]. This is because of the existence of large amounts of outliers, which most of the graph matching algorithms do not take into account [10, 11, 12]. This fact is also reported for many real world scenarios where higher order similarities that employ contextual information have been proposed as a solution [13]. We have also experienced quite similar phenomena with pairwise measurements in the presence of outliers. This fact will be presented in the experimental study in Section 5 (symbol spotting experiment) as a benefit of the proposed contextual similarities (see Figure 8).

The main motivation of this work is the use of contextual information of nodes (i.e. neighbouring structures) to make subgraph matching more robust and efficient in large scale visual recognition scenarios. A second key component is the formulation of subgraph matching with approximate algorithms. Therefore, in this paper we propose an inexact subgraph matching methodology based on tensor product graph (TPG). Given two attributed graphs it is quite straight forward to get the pairwise similarities and assign them as weights on the edges of TPG (step one in Figure 1). Next one can think of having a random walk from node to node considering the weights on the edges as the probabilities to proceed to the next node. Finally, we accumulate the probabilities of finishing a walk at each of the vertices, which we refer to as contextual similarities (CS), where the context of each node is the set of its neighbouring nodes. This procedure essentially diffuse the pairwise similarities in the context of neighbouring nodes. This information can be obtained by simple algebraic operation of the adjacency (or weight) matrix of the product graph (step two in Figure 1). A similar phenomenon is termed diffusion on tensor product graph, which is well known to capture the geometry of data manifold and higher order contextual information between objects [14, 13]. We formulate subgraph matching as a node and edge selection problem in TPG. To do that we use those contextual similarities and formulate a constrained optimization problem (COP) to optimize a function constructed from the higher order similarity values (step three in Figure 1). We solve the optimization problem with linear programming (LP) which is solvable in polynomial time. Depending on whether we need a discrete solution or not, we may add a discretization step. In this paper we will show that the higher order contextual similarities allow us to relax the constrained optimization problem in real world scenarios.

The main contribution of this work is to propose a random walk based approach to obtain contextual similarities between nodes that capture higher order information. In the literature, there are methods that formulate graph matching explicitly using higher order information of nodes and edges[15, 16]. They often embed the higher order relations of nodes by hypergraph matching formulations. Usually the affinity of feature tuples are modelled using tensor and later approximate algorithms are proposed to obtain the correspondences [15]. Our work takes into account the higher order context information by propagating the pairwise similarities with random walk formulation. The procedure counts the number of incoming walks to the nodes of TPG. Here the general hypothesis is that a particular node in TPG, that constitutes a pair of nodes of the operand graphs likely to be matched, should receive a reasonably high amount of random walks. This idea is also supported by other works [17, 14, 18, 15, 13, 19], where the idea of random walk has been used for obtaining similarity of pairs of nodes. Yet, our formulation and interpretation are completely different than the others. We have used the classical algebraic formulation to obtain the similarities between nodes whereas most of the previously mentioned algorithms have used an approximated iterative procedure to obtain the similarities or the node to node correspondences depending on the requirement. Apart from that, we have revealed that backtrackless walks [20] can also be utilised for obtaining contextual similarities of pairs of nodes. Later, we use these contextual similarities to formulate a subgraph matching procedure as a node and edge selection procedure in TPG. This is an optimization problem and here we are motivated by the ILP formulation of Le Bodic et al. [21]. We have proved the effectiveness of the proposal with a detailed experimental study.

The rest of the paper is organized in four sections. In Section 2, we give a formal definition of graph matching and present the state-of-the-art of (sub)graph matching algorithms. We present a product graph based subgraph matching methodology which includes the description of the contextual similarities in Section 3 and the formulation of subgraph matching as an constrained optimization problem in Section 4. In Section 5 we present experimental results showing the robustness of the method. At last in Section 6 we conclude the paper and outline possible future directions of this work.

2 Previous work on graph matching

2.1 Formal definition of graph matching

Let $G_{1}(V_{1},E_{1},\alpha_{1},\beta_{1})$ and $G_{2}(V_{2},E_{2},\alpha_{2},\beta_{2})$ be two attributed graphs with $|V_{1}|=n_{1}$ and $|V_{2}|=n_{2}$ . Let $\kappa^{v}_{i_{1}i_{2}}$ measures the affinity between the nodes $u_{i_{1}}\in V_{1}$ , $u_{i_{2}}\in V_{2}$ and $\kappa^{e}_{i_{1}i_{2},j_{1}j_{2}}$ measures the affinity between the edges $(u_{i_{1}},v_{j_{1}})\in E_{1}$ and $(u_{i_{2}},v_{j_{2}})\in E_{2}$ . Given these two graphs together with node and edge affinity measures, one can compute the affinity matrix $\mathbf{K}\in\mathbb{R}^{n_{1}n_{2}\times n_{1}n_{2}}$ in a concise way as follows:

[TABLE]

Using $\mathbf{K}$ , graph matching can be concisely formulated as the following quadratic assignment problem (QAP):

[TABLE]

where $\mathbf{X}\in\Pi$ is constrained to be a one-to-one mapping, i.e. $\Pi$ is the set of partial permutation matrices:

[TABLE]

Solving Eqn. (2) in an exact way leads to IQP which is a typical NP-hard problem.

2.2 Related works

Very extensive and detailed reviews of all kind of graph matching methods for pattern recognition is available in [22, 23, 24]. In pattern recognition, the research community has divided the graph matching methods into two broad categories: (1) exact and (2) inexact graph matching. Exact matching requires a strict correspondence between the two objects being matched or at least between their sub parts [25, 26, 27]. However, the strong constraints of the exact matching algorithms are too rigid for real applications in computer vision. This is because the description of an object (or part of it) represented as the node/edge attributes of a graph is a vector resulting from some statistical analysis. For that reason researchers proposed inexact graph matching which allows some tolerance under a similarity model and in this case a matching can occur when two graphs being matched are structurally different to some extent. Among the earlier works, a group of methods worked with tree search with backtracking [28, 29]. These methods gave birth of the popular graph edit distance. In these papers the authors introduced the graph edit costs i.e. the costs for substituting, deleting, inserting and merging nodes and edges to define a distance between two graphs [30, 31]. Later, graph edit distance was shown to be effective for explicit graph embedding [32] and graph kernel [33], although detailed discussions on these topics are out of the scope of the article.

Most of the modern methods address the inexact matching problem and focus on the relaxation of permutation constraints (Eqn. (3)). Based on the approximation strategy and the applied methodology, these methods can be broadly classified into three categories: (1) spectral decomposition, (2) semidefinite programming (SDP) relaxation and (3) convex relaxation.

2.2.1 Spectral decomposition

The first set of methods estimates the graph permutation matrix $\mathbf{X}$ with an orthogonal one, i.e., $\mathbf{X}^{T}\mathbf{X}=\mathbf{I}$ . Under this orthogonal constraint, optimizing graph matching can be solved as an eigenvalue problem [10, 34]. These methods often suffer when the spectral embeddings of the adjacency matrices are not uniquely defined. The problems in computer vision are much more complex and to deal with them several researchers have proposed a wealth of effective and efficient solutions [35, 36].

2.2.2 Semidefinite programming

Semidefinite programming is a general relaxation tool for approximating many combinatorial problems. Consequently, it has largely been used to approximate the graph matching problem. These methods usually redefine the objective of graph matching by introducing a new variable $\mathbf{Y}\in\mathbb{R}^{n_{1}n_{2}\times n_{1}n_{2}}$ such that $\mathbf{Y}=\text{vec}(\mathbf{X})\text{vec}(\mathbf{X})^{T}$ . Here semidefinite programming (SDP) approximates the non-convex constraint on $\mathbf{Y}$ as a semidefinite one so that it satisfies $\mathbf{Y}-\text{vec}(\mathbf{X})\text{vec}(\mathbf{X})^{T}\geq 0$ . The main advantage of the SDP methods is their theoretical assurance to obtain a solution in polynomial time. After computing $\mathbf{Y}$ using SDP, a probabilistic post-processing step is included to get a discrete solution [37, 38]. Nonetheless, sometime SDP algorithms are rather expensive as the size of $\mathbf{Y}$ is quadratic to the actual problem size.

2.2.3 Convex relaxation

Most of the other methods relax $\mathbf{X}\in\mathcal{D}$ to be a doubly stochastic matrix, the convex hull of the set of permutation matrices:

[TABLE]

Presuming this constraint, optimizing graph matching can be considered as a convex quadratic programming problem. Various methods have already been proposed to find a local optimum. One of the first approaches along this line was proposed by Almohamad and Duffuaa [39], where the authors transformed the quadratic optimization problem to a linear one and solved it with a combination of linear programming and the Hungarian algorithm. Gold and Rangarajan [40] proposed a graduated assignment algorithm by combining graduated nonconvexity, two-way assignment constraints, and sparsity. The method iteratively solves a series of linear approximations of the cost function using Taylor’s series expansion to find a fast and accurate solution. Pelillo et al. [41] proposed a series of algorithms that mainly search for a maximal clique in an association graph of two operand graphs for their matching. They utilised the replicator dynamics equation to solve the problem of maximal clique finding in an approximate way. Van Wyk and Van Wyk [42] proposed an algorithm to iteratively project the approximate correspondence matrix onto the convex domain. Similar to [40], they enforced two-way assignment constraints but did not use elaborated penalty terms, graduated nonconvexity, or sophisticated annealing mechanisms to avoid poor local minima. Torresani et al. [43, 44] designed a complex energy function depending on the appearance, geometry and spatial coherence of the features to solve the matching task. They proposed a dual decomposition based approximate algorithm to optimize the energy function. Zass and Shashua [45] presented a convex relative entropy error model emerged from a probabilistic interpretation of the graph matching problem. Leordeanu et al. [46] proposed an integer projection algorithm with climbing and convergence properties to optimize the objective function in an integer domain. They claimed their algorithm gives excellent results starting from the solution returned by any graph matching algorithm. Zaslavskiy et al. [11] proposed a path following algorithm of a convex-concave problem obtained by linear interpolation of convex and concave formulations, starting from the convex relaxation. Inspired by the PageRank algorithm [47], Cho et al. [18] proposed a random walk based iterative procedure for the graph matching problem. As an extension to [11], Liu et al. [48] proposed another path following strategies for the non-convex quadratic programming problem. Their algorithm takes into account the directed graph whereas the algorithm proposed by Zaslavskiy et al. [11] considered undirected graphs. Their algorithm is also applicable for getting the discrete solution from the continuous solution of any other graph matching algorithm. Zhou and De la Torre [49, 50] presented a novel framework based on the factorization of the pairwise affinity matrix which provides several benefits from the graph matching point of view.

Apart from formulating better approximations for Eqn. (2), there are two more directions of graph matching methods that reported improved results: (1) learning the pairwise affinity matrix [51, 52] and (2) explicitly incorporating higher order features [45, 15, 16]. The first set of methods learns in a supervised or unsupervised way the compatibility functions such that the solution of the resulting graph matching problem best matches the expected solution that a human would manually provide. A slightly different approach is to learn class specific graph models to match with target graphs [53]. The second category of methods addresses the problem of establishing correspondences between two sets of visual features using higher order constraints instead of the unary or pairwise ones [15, 16]. By imposing higher order constraints, the authors formulate graph matching as a hypergraph matching task which is formulated as the optimization of a multilinear objective function defined by a tensor representing the affinity between feature tuples. There are methods that work with equal sized graphs [11, 54]. But most of the modern methods can deal with graphs of different size, though usual graph matching algorithms perform worse in subgraph matching scenario [13].

Despite having many methods for graph matching available, very few of them explore the fact of context for strengthening node-node matching affinities [18]. Also many of the above methods fail to perform well in presence of a high number of outliers. To solve these problems, in this paper, we propose contextual similarities on the tensor product graph and use them for subgraph matching.

3 Contextual similarities on product graph

A tensor product graph (TPG) $G_{X}(V_{X},E_{X},p_{X},W_{X})$ of two operand graphs $G_{1}(V_{1},E_{1},\alpha_{1},\beta_{1})$ and $G_{2}(V_{2},E_{2},\alpha_{2},\beta_{2})$ is a graph such that the vertex set $V_{X}$ is the Cartesian product $V_{1}\times V_{2}$ i.e.

[TABLE]

and the edge set $E_{X}$ is defined as:

[TABLE]

and $p_{X}$ and $W_{X}$ contain weights as a function of $\alpha_{1}$ , $\alpha_{2}$ and $\beta_{1}$ , $\beta_{2}$ , respectively.

A very interesting property of the adjacency matrix $A$ of any graph $G$ is that the $(i,j)$ th entry of $A^{n}$ denotes the number of walks of length $n$ from node $v_{i}$ to node $v_{j}$ . One can bring the same analogy to an edge weighted graph where each edge is associated with weights in $[0,1]$ . Let $W$ be such a matrix that contains weights on the edges. These weights can be considered as the probabilities (after normalization, when each of its columns sums to one) of a random walker moving from one node to another. Then following the same idea the $(i,j)$ th entry of $W^{n}$ denotes the probability of having a walk of length $n$ from the node $v_{i}$ to the node $v_{j}$ . Now, let $p_{0}$ be the stopping probability distribution over vertices ( $p_{0}$ sums to one), then the probability distribution $p_{k}$ describing the location of the random walker at the end of $k$ th transition is $p_{k}=p_{0}\odot W^{k}\mathbf{1}$ [55]. Here a higher entry reveals that more and more random walks finish at a particular node and this particular node is better connected with rest of the graph.

The same procedure can be simulated on a TPG of two operand graphs and can be used to capture higher order contextual information. The process can be started by assigning the pairwise similarities of the edges as the weights on the corresponding edges of TPG. Let $W_{X}$ be such a matrix, where each of its columns sums to one (with weights converted to probabilities). This matrix basically contains the probabilities of transition of a random walker from a node to the next one. Let $p_{X}$ be the stopping probability distribution over the vertices. Practically this is a vector which contains the node to node similarities that sum up to one. Then simultaneous walking can be performed from node to node taking these probabilities to move from one node to the next one. Similar to the above explanation, the position of the random walker, after finishing a $k$ -length walk, can be obtained by a probability distribution $p_{X_{k}}=p_{X}\odot W_{X}^{k}\mathbf{1}$ . The $i$ th component of $p_{X_{k}}$ denotes the probability of finishing a $k$ -length walk at the vertex enumerated $i$ .

Let

[TABLE]

Similar to the previous explanation, here the $i$ th component of $S_{p_{X}}$ indicates the likelihood of finishing a walk at node $v_{i}$ and a higher entry reveals that the node is better connected with rest of the TPG. As the weights on the edges of TPG come from the similarities of edges of the operand graphs, here a better connected node of TPG is supposed to constitute a promising pair of matched nodes. Since the walking is performed through the edges of TPG, the accumulated weights take into account contextual information. This procedure of accumulating weights by considering connections between objects is proved to be more discriminative for objects where contextual information is important. This is also the intrinsic idea of graph diffusion, which is well known to capture higher order context similarities and complex relations. It is observed by several recent approaches that when pairwise similarities are diffused in the context of other neighbouring points, more reliable similarities are obtained [17, 14, 18, 15, 13, 19]. In other words, this random walk procedure reevaluates the pairwise similarities within the context of neighbouring nodes, resulting in more robust affinities that we termed contextual similarities. In our graph matching module, we use these reevaluated similarity values, which are more robust than pairwise similarities. In this way our method is more advantageous than other existing graph matching methods based on random walk [18]. In this paper, we model the procedure of walking in two different ways, viz., random walks and backtrackless walks which we describe below.

3.1 Random walks

The easiest way to get the contextual similarities between pairs of nodes through a graph is by propagating the pairwise similarity information with random walks on TPG. Below we use $\mathbf{I}$ to denote the identity matrix. When it is clear from context we will not mention the dimensions of any of the vectors or matrices. The process of obtaining contextual similarities with random walks can be defined as:

[TABLE]

where

[TABLE]

Here $\lambda$ is a weighting factor to discount the longer walks, as they often contain redundant or repeated information such as cycles. In this paper we always choose $\lambda=\frac{1}{a}$ , where $a=\min(\Delta^{+}(W_{X}),\Delta^{-}(W_{X}))$ . Here $\Delta^{+}(W_{X})$ , $\Delta^{-}(W_{X})$ are respectively the maximum outward and inward degree of $W_{X}$ [56]. The infinite summation of Eqn. (6) converges to $(\mathbf{I}-\lambda W_{X})^{-1}$ for sufficiently small values of $\lambda$ .

Eqn. (6) can be efficiently computed by solving $(\mathbf{I}-\lambda W_{X})x=\mathbf{1}$ by conjugate gradient methods which allows to avoid the expensive matrix inversion and then performing Hadamard product by $p_{X}$ . An entry $W^{CS}_{X}(\omega_{i},\omega_{j})$ indicates the likelihood of finishing a random walk at the node $(\omega_{i},\omega_{j})$ of TPG. Since the weights on the edges of TPG are derived from the pairwise similarities of edge attributes, a higher value in $W^{CS}_{X}(\omega_{i},\omega_{j})$ reveals a better candidate for pairwise matching. Here it is to be noted that $W^{CS}_{X}$ is a column vector, the comma separated double subscript is used just to ease the understanding.

For the sake of understandability, let us illustrate the idea using a simple example. Let us consider an edge similarity matrix $W$ (Eqn. (8)) and node similarity vector $p$ (Eqn. (9)) as follows:

[TABLE]

Here the contextual similarities obtained after finishing the 2-length walks are

[TABLE]

The rows of the matrix on RHS of Eqn. (10) show the contextual similarities of respective nodes. As the entries of $W$ contain the edge similarities, it is clear that the procedure takes into account the similarities of the walks of different length (walks upto length two in this example) to respective nodes. Hence it is evident that the exponentiation plus summation procedure takes into account information from the context to determine the strength of a particular node. An actual occurrence of a pattern graph in the target graph creates higher similarities in the neighbourhood, which clearly effects the connected nodes. This formulation enhances the pairwise similarities with more information from context.

The main drawback of the random walk based procedure is that it backtracks an edge in case of a undirected graph and this reduces its discriminatory power. To solve this limitation, backtrackless walks, a variation of random walks have been recently proposed [20]. In the next section we describe how to adapt this idea to our approach.

3.2 Backtrackless walks

A similar formulation as random walks can be used with backtrackless walks [20]. The backtrackless walks are random walks but do not backtrack an edge and for that a variation of exponentiation of the weight matrix is available [57]. The main advantage of backtrackless walks over random walks is that they avoid tottering, and hence increase the discriminative power of the resulting contextual similarities. The benefit of contextual similarities generated through this type of walks has been corroborated in our experiments on undirected graphs (see Section 5.2 and Section 5.4). The process of obtaining contextual similarities with backtrackless walks can be defined as (proof is provided in the supplemental material):

[TABLE]

where

[TABLE]

Here $Q_{X}$ is a diagonal matrix where the $i$ th or $(i,i)$ th element is equal to the $i$ th or $(i,i)$ th element of $W_{X}^{2}$ minus one. Parameter $\lambda$ serves the same purpose as before. The summation in Eqn. (11) converges for sufficiently small value of $\lambda$ . Similar to Eqn. (6), Eqn. (12) can also be computed by solving $(\mathbf{I}-\lambda W_{X}+\lambda^{2}Q_{X})x=p_{X}$ and then multiplying the solution by $(1-\lambda^{2})$ .

Here the phenomena regarding context can be explained with the same example as in Eqn. (8), Eqn. (9). According to that example, $Q_{X}$ can be written as follows:

[TABLE]

Now contextual similarities obtained after finishing the 2-length walks are:

[TABLE]

Here also it is clear that the exponentiation plus the summation procedure takes into account contextual information to determine the strength of a particular node. In each iteration it eliminates the tottering effect (by eliminating the loops) with the special algebraic formulation in Eqn. (13). An actual occurrence of a pattern graph in the target graph creates higher pairwise similarities in the neighbourhood, and this also effects the connected nodes. This formulation enhances the pairwise similarities with more information from context/connection. To the best of our knowledge, this work is the first to explore the contextual similarities obtained by backtrackless walks. The contextual similarities obtained by these procedures are based on counting the common random or backtrackless walks in the operand graphs. As the edges of the product graph encode the similarity based on the features that are invariant to transformation, rotation and scale, contextual similarities obtained by these procedures are also invariant to such geometric transformation.

We use the contextual similarities obtained in the steps explained above to formulate a subgraph matching algorithm as a constrained optimization problem.

4 Subgraph matching as a constrained optimization problem

The contextual similarities obtained in the previous step just provide node-node, edge-edge affinities and do not yield correspondences. We formulate subgraph matching as a node and edge selection problem in tensor product graph. To do that we use the contextual similarities obtained in the previous step to construct a maximization problem and solve it with a linear programming formulation. Let

[TABLE]

and

[TABLE]

Here $\Delta^{+}(u_{i},u_{j})$ denotes the maximum outward degree of the node $(u_{i},u_{j})\in V_{X}$ . Here it is to be clarified that both $S_{V}$ and $S_{E}$ are vectors, the comma separated double subscripts are just to ease understanding. Now clearly the dimension $S_{V}$ is $|V_{X}|$ and that of $S_{E}$ is $|E_{X}|$ . We formulate the subgraph matching problem as a node and edge selection problem in the product graph. This can be regarded as a constrained optimization problem which maximize a function of higher order similarities of the nodes and edges. We construct the objective function as follows:

[TABLE]

where $x$ and $y$ are vectors containing variables denoting the probabilities of matching (selecting) the nodes and edges in the product graph respectively. For example, $x_{u_{i},u_{j}}$ denote the probability of matching node $u_{i}\in V_{1}$ with node $u_{j}\in V_{2}$ . Similarly, $y_{u_{i}u_{j},v_{i}v_{j}}$ denote the probability of matching the edge $(u_{i},v_{j})\in E_{1}$ with edge $(u_{j},v_{j})\in E_{2}$ . Both $x$ and $y$ contain probabilities, as the optimization problem in Eqn. (16) is solved with linear or continuous programming with the domain $[0,1]$ . To get a discrete values of the solution we apply the Hungarian algorithm on the node to node matching probabilities (here $x$ ). It is to be mention that the above objective function in Eqn. (16) can easily be written in the form of Eqn. (2), where $\mathbf{K}$ is given as follows:

[TABLE]

Now let us introduce a set of constraints on the variables to satisfy the maximal common subgraph matching problem between the operand graphs $G_{1}$ and $G_{2}$ in TPG $G_{X}$ .

Pattern node constraint Each node $u_{i}\in V_{1}$ can be matched at most with one node $u_{j}\in V_{2}$ i.e. there can be at most a single node $(u_{i},u_{j})\in V_{X}$ for each $u_{i}\in V_{1}$ .

[TABLE]

2.

Pattern edge constraint Each edge $(u_{i},v_{i})\in E_{1}$ can be matched at most with one edge $(u_{j},v_{j})\in E_{2}$ i.e. there can be at most a single edge $((u_{i},u_{j}),(v_{i},v_{j}))\in E_{X}$ for each $(u_{i},v_{i})\in E_{1}$

[TABLE]

3.

Target node constraint Each node $u_{j}\in V_{2}$ can be matched with at most one node in $u_{i}\in V_{1}$ i.e. there can be at most one node $(u_{i},u_{j})\in V_{X}$ for each $u_{j}\in V_{2}$ .

[TABLE]

4.

Target edge constraint Each edge $(u_{j},v_{j})\in E_{2}$ can be matched with at most one edge $(u_{i},v_{i})\in E_{1}$ i.e. there can be at most one edge $((u_{i},u_{j}),(v_{i},v_{j}))\in E_{X}$ for each $(u_{j},v_{j})\in E_{2}$

[TABLE]

5.

Outward degree constraint The outward degree of a node $(u_{i},u_{j})\in V_{X}$ is bounded above by the minimum of the outward degree of the node $u_{j}\in V_{1}$ and $u_{j}\in V_{2}$ i.e. the number of outgoing edges from the node $(u_{i},u_{j})\in V_{X}$ is less than or equal to the minimum of the outward degree of the nodes $u_{i}\in V_{1}$ and $u_{j}\in V_{2}$ .

[TABLE]

6.

Inward degree constraint The inward degree of a node $(v_{i},v_{j})\in V_{X}$ is bounded above by the minimum of the inward degree of the node $v_{i}\in V_{1}$ and $v_{j}\in V_{2}$ i.e. the number of incoming edges to the node $(v_{i},v_{j})\in V_{X}$ is less than or equal to the minimum of the inward degree of the nodes $v_{i}\in V_{1}$ and $v_{j}\in V_{2}$ .

[TABLE]

7.

Domain constraint Finally, we restrict all the variables to be in between $[0,1]$ .

[TABLE]

The procedure of obtaining contextual similarities using random walks has some resemblance with the PageRank algorithm [47], power iteration method for eigenvalue problem [16] or some other similar algorithms [18]. But their further utilization and the idea of backtrackless walks instead of simple random walks were not done before.

5 Experimental results

In this section, we report four different experiments on different benchmark datasets and provide a comparison of the proposed algorithms with several state-of-the-art methods. In the first three experiments, we examine the performance of graph matching algorithms. We compare with eleven state-of-the-art algorithms: (1) Graduated assignment (GA) [40], (2) Spectral matching (SM) [35], (3) Spectral matching with affine constraints (SMAC) [36], (4) Integer projected fixed point method (IPFP-U) [46], (5) IPFP-S [46], (6) Probabilistic matching (PM) [45], (7) Reweighted random walks matching (RRWM) [18], (8) Factorized graph matching (FGM-D) [49], (9) FGM-U [49], (10) Dual decomposition (DD) [44], (11) High-order graph matching (HOGM) [16]. Here IPFP-U and IPFP-S are basically the same algorithms; they only differ in the initialization procedure. Similary, FGM-U and FGM-D are also the same but designed for undirected and directed graphs, respectively. We have used some acronyms for referring to our algorithms as follows: (1) the algorithm working with pairwise similarities as PG-N, (2) the algorithm working with contextual similarities based on random walks as PG-R and (3) the algorithm working with contextual similarities based backtrackless walks as PG-B. For these experiments, we computed the matching accuracy (Eqn. (17)) and normalized objective score (Eqn. (18)) for evaluating and comparing the performance. The matching accuracy is defined as the number of consistent matches between the corresponding matrix $\mathbf{X}_{algthm}$ given by an algorithm and ground truth $\mathbf{X}_{truth}$ :

[TABLE]

The objective score of an algorithm is defined as the objective value obtained corresponding to the consistent matches obtained by that algorithm:

[TABLE]

The normalized objective score is the objective scores scaled in $[0,1]$ .

The fourth experiment is more focused on an application of inexact or error tolerant subgraph matching. Here we convert the symbol spotting problem in graphical documents to a subgraph matching problem and apply the proposed product graph based subgraph matching algorithm for spotting symbols. The performance of the proposed algorithm in presence of a large number of outliers is tested.

All the experiments were done on a workstation with Intel i5 2.50GHz processor and 8GB of RAM222Our unoptimized Matlab code that we had used for the experimentations is available in https://sites.google.com/site/2adutta/research..

5.1 Synthetic dataset

This particular experiment performed a comparative evaluation of different graph matching algorithms on randomly generated synthetic graphs following the experimental framework shown in [40]. This is a common experimental framework and allows one to test graph matching algorithms under various parameter settings. For that, in each run, we constructed two different graphs $G_{1}(V_{1},E_{1},\alpha_{1},\beta_{1})$ and $G_{2}(V_{2},E_{2},\alpha_{2},\beta_{2})$ each of which contained 20 inlier nodes. In each run, we added $n_{out}$ number of outlier nodes to both graphs. Each node $u_{i}\in V_{1}$ was assigned with a uniformly distributed node label $\alpha_{1}(u_{i})\backsim\mathcal{U}(0,1)$ and the corresponding node $u_{j}\in V_{2}$ was disturbed according to $\alpha_{2}(u_{j})=\alpha_{1}(u_{i})+\epsilon$ by adding a Gaussian noise $\epsilon\backsim\mathcal{N}(0,\sigma^{2})$ . For each pair of nodes, an edge was randomly created depending on the edge density parameter $\rho\in[0,1]$ . Each edge $(u_{i},v_{i})\in E_{1}$ was assigned with a uniformly distributed edge label $\beta_{1}(u_{i},v_{i})\backsim\mathcal{U}(0,1)$ and the corresponding edge $(u_{j},v_{j})\in E_{2}$ was disturbed as $\beta_{2}(u_{j},v_{j})=\beta_{1}(u_{i},v_{i})+\epsilon$ by adding a Gaussian noise $\epsilon\backsim\mathcal{N}(0,\sigma^{2})$ . Here it is to be noted that this kind of edge labelling created a directed graph. The node affinity matrix $p_{X}$ was computed as $p_{X}=\exp(-\frac{(\alpha_{1}(u_{i})-\alpha_{2}(u_{j}))^{2}}{0.15})$ and the edge affinity matrix $W_{X}$ was computed as $W_{X}=\exp(-\frac{(\beta_{1}(u_{i},v_{i})-\beta_{2}(u_{j},v_{j}))^{2}}{0.15})$ .

This experiment tested the performance of graph matching algorithms under three different parameter settings. For each setting, we generated 100 different pairs of $G_{1}$ and $G_{2}$ which resulted in 100 independent runs and evaluated the average accuracy (Eqn. (17)) and normalized objective score (Eqn. (18)) over these 100 runs. In the first setting, we varied the number of outliers ( $n_{out}$ ) 0:2:20 and fixed $\rho=1$ and $\sigma=0$ . In the second parameter setting, we varied the noise parameter ( $\sigma$ ) 0:0.02:0.2 and fixed $n_{out}=0$ and $\rho=1$ . In the last setting, we varied the edge density parameter ( $\rho$ ) 1.0:-0.1:0.2 and fixed $n_{out}=0$ and $\sigma=0$ . In the first two parameter settings, our proposed methods, PG-N, PG-R and PG-B, achieved the best performance together with RRWM, FGM-D in terms of accuracy and objective ratio (see Figure 2(a),(b),(d),(e)); although DD and HOGM performed quite closely to the previously mentioned algorithms in these two parameter settings as well. In the last parameter setting, together with most of the other methods, the performance of our algorithms deteriorates as the edge density decreases. This is an obvious phenomenon as any decrement of the edge density destroys the structure of graphs and hence loses the representation power. Our contextual similarities are obtained by propagating the pairwise similarities through the graph edges, which lose discrimination in the absence of edges. This explains the worse performance of our algorithm in this parameter settings. Despite having sparse graphs, DD and HOGM performed quite well as these two algorithms use node affinity, their local distribution in 2-dimensional space and higher order structures as features for matching. In general, the time to find a solution of all the considered algorithms usually increases as the number of outliers and the amount of deformation increases. Nevertheless, the execution time of different algorithms behaves differently as the edge sparsity increases. For example, HOGM performed more efficiently with a decreasing edge density whereas FGM-U performed completely opposite to HOGM. Our algorithms and FGM-D showed slightly different time efficiency. Initially, with the increment of edge sparsity, their time duration increases to get an effective solution. After a certain point it decreases and also the accuracy drops accordingly.

5.2 House and Hotel image dataset

The CMU house and hotel image sequences333http://vasc.ri.cmu.edu//idb/html/motion are among the most popular datasets to test the performance of graph matching algorithms [40, 51, 18, 52, 49]. This dataset provides a more realistic way to judge the performance of graph matching algorithms. It consists of 111 frames of a toy house and 101 frames of a toy hotel, each of which had been manually labelled with 30 landmark points. Each individual frame in these sequences has a certain difference in the position of viewpoint with respect to its adjacent frames. We used Delaunay triangulation to connect the landmarks. Each node was assigned with a shape context descriptor. The edge label $\beta$ was computed as the length of the edge, here it is to be mentioned that this kind of edge labelling created undirected edges. In this experiment both the node and edge affinities were computed as dot product similarities. We performed the experiment with all possible image pairs, spaced by 0:10:90 frames. To evaluate the performance we computed the average accuracy (Eqn. (17)) and normalized objective score (Eqn. (18)) in each sequence gap.

For both sequences, we tested the performance of graph matching methods under two scenarios. In the first case, we used all the 30 nodes (annotated landmark points) and in the second one we randomly chose 25 nodes from both the graphs and performed a subgraph matching. It can be seen that for the house sequence in the first case, IPFP-U, IPFP-S, RRWM, FGM-U, FGM-D, HOGM and PG-B obtained almost perfect matchings of the original graphs (see Figure 3(a)-(b)). But for the second case, as some of the nodes from both the graphs got hidden and graphs got corrupted, performance of all the methods deteriorated (see Figure 3(d)-(e)). Still our method PG-B performed best among all. Here the difference in performance between the method working with backtrackless walk (PG-B) and the method working with random walk (PG-R) is to be noticed. Since the graphs considered for this experiment are undirected, removal of tottering proves to be effective. Nevertheless, the difference in performance between the method working with pairwise similarities (PG-N) and the methods working with contextual similarities (PG-R, PG-B) is to be noticed. Since both PG-R and PG-B are taking into account contextual information they performed better than PG-N. The hotel sequence appeared as a harder challenge to all the algorithms, as most of them performed worse than the house sequence. Here all the methods performed worse with distorted graphs (see Figure 3(j)-(k)) than their undistorted versions (see Figure 3(g)-(h)). Here also in both cases our methods with contextual similarities performed best. Qualitative results obtained by PG-B on this dataset are shown in Figure 4. In case of an undistorted pair of graphs, the execution time of most of the methods increases with the baseline. In difficult cases such as increased baseline, DD takes a lot of iterations to converge. This explains its significant increase of execution time as well as the poor results with the increment of baseline.

5.3 Car and Motorbike image dataset

The car and motorbike image dataset444https://goo.gl/CCp9lA was created in [52]. This dataset consists of 30 pairs of car images and 20 pairs of motorbike images that were taken from the dataset of PASCAL challenges. Each pair of images consists of $15\backsim 52$ true correspondences. Since these graphs contain a rather large number of outlier nodes, this experiment provides a harder challenge. The node label $\alpha$ was computed as the orientation of the normal vector to each of the contours. Like before, we adopted Delaunay triangulation to build the graph. For this experiment, edge label $\beta$ was computed as the edge lengths and angles with the horizontal. Due to the nature of the edge labels, the graphs constructed were directed.

We tested the performance of the graph matching methods against noise and for that we added $0:2:20$ outlier nodes to both graphs. For performance evaluation we computed the average accuracy (Eqn. (17)) and normalized objective score (Eqn. (18)) within a level of outliers (see Figure 6). The performance of the algorithms is quite similar. Particularly, FGM-D, PG-R and PG-B were quite competitive and obtained similar performance. In both cases, PG-N performed worse than PG-R and PG-B due to the same reason as explained previously. As in this case the graphs considered were directed, very minor difference in performance was observed between PG-R and PG-B. Qualitative results obtained by PG-B on this dataset are shown in Figure 5.

5.4 Symbol spotting as an inexact subgraph matching problem

Symbol spotting is a well known problem in graphic recognition which is a subfield of document image analysis. It can be defined as locating a given query symbol in large graphical documents such as line drawings, floorplan, circuit diagrams etc. As line drawings have especial structural and syntactic nature, symbol spotting is often devised as subgraph matching problem [1, 58, 21]. Due to the existence of adequate background information, structural noise, large target graphs, this experiment presents a hard challenge to a subgraph matching algorithm. We considered the SESYD dataset555http://mathieu.delalandre.free.fr/projects/sesyd [59] for this experiment. The original SESYD dataset is a collection of 10 different subsets of floorplan images (shown in Figure 7a), where each of the sets is created by putting different isolated symbols (see Figure 7b-Figure 7q) in different orientations, scales etc. For this particular experiment we randomly selected 20 images from each of the 10 subsets which in total resulted in 200 images. This experiment provides a complete framework for inexact subgraph matching from application point of view.

For representing these graphical documents we considered the dual edge graph representation proposed in [60]. Let $G_{1}=(V_{1},E_{1},\alpha_{1},\beta_{1})$ and $G_{2}=(V_{2},E_{2},\alpha_{2},\beta_{2})$ be respectively the attributed dual edge graphs for the pattern and the target. The target graphs in this dataset contain far more outlier nodes than any of the datasets described before. Here $\alpha:V_{1}(u)\rightarrow\mathbb{R}^{m\times 7}$ is a node labelling function and is defined as the set of seven Hu moments invariants [61] of the acyclic graph paths joining the extremities of an edge in the edge graph ( $m$ is total number of paths). $\beta:E_{1}(u_{i},v_{i})\rightarrow\mathbb{R}^{3}$ is an edge labelling function and is constructed of three components:

normalized angle (angle divided by $180^{\circ}$ ) between two edges (of the edge graph) joined by the dual edge.

2.

the ratio of the two edges (of the edge graph) joined by the dual edge.

3.

the ratio of the distance between the mid points of the two edges (of the edge graph) and the total length of the edges joined by the dual edge.

For computing the distance between the dual nodes we used a modified version of Hausdorff distance [62] as follows:

[TABLE]

Here $dm(a,b)$ denotes the Euclidean distance of $a\in A$ from $b\in B$ and $d(A,B)$ denotes the distance between the sets $A$ and $B$ . Since the node label in our dual edge graph representation is a set of Hu moments, we used this particular distance measure. Then the affinities were obtained as $\exp{(-d)}$ . For having the distance between the edges we used the Euclidean distance and here also their similarity were obtained as $\exp{(-d)}$ .

In Figure 8, we have shown an example of node-node matching for three different scenarios. In that figure, the left sub figure shows the correspondences obtained by PG-N, the one in the middle shows the correspondences obtained by PG-R and the one in the right shows the same with PG-B. Here it is clear that the contextual similarities really enhance the truly matched nodes, as in the case of random, backtrackless walks the most of the obtained correspondences focus to the true matchings. After obtaining the correspondences, we performed a simple clustering to group the adjacent dual nodes to obtain a bounding box for comparing with the ground truth. A single bounding box is supposed to contain a single instance of a pattern graph.

The decision whether a retrieved subgraph is true or false was done by overlapping the bounding boxes. This signifies: if $B$ be the rectangular bounding box for a retrieval and $T$ be the bounding box of its corresponding ground truth, $B$ is considered true if $\frac{B\cap T}{B\cup T}\geq 0.5$ . For quantitative evaluation, we computed the usual metrics for evaluating a retrieval system, such as precision (P), recall (R), F-measure (F) and average precision (AP). For detailed definitions of all such metrics, we refer to the paper [63]. To get an idea about the time complexity we have also shown the meantime (T) of matching a pattern graph in a target graph. To have a comparison among the state-of-the-art symbol spotting methods, we considered five different methods. They are: (1) Symbol spotting with graph representation (SSGR) [64], (2) Integer linear programming (ILPIso) [21], (3) Fuzzy graph embedding (FGE) [65], (4) Serialized graphs (SG) [8], (5) Near convex region adjacency graph (NCRAG) [66].

The quantitative results obtained by different methods are listed in Table 1 and the precision-recall curves are shown in Figure 9a. From the results it is clear that the methods working with contextual similaritis (PG-R and PG-B) outperformed all the other methods. Moreover, there is a difference observed in the performance of PG-R and PG-B, and this phenomenon can be explained as a benefit of PG-B in the case of undirected graphs which eventually eliminates the totterings in the graph edges. Like most of the methods, our methods showed comparatively low performance for sink3 (Figure 9b) and door1 (Figure 9c). This is because of the existence of sink2 and door2, which respectively contain the previously mentioned pattern graphs. Also worse performance was obtained for sofa1 (Figure 9d) and this happened as this pattern frequently occurred in floorplans.

A qualitative result666Rest of the qualitative results are available on https://sites.google.com/site/2adutta/research is shown in Figure 10, where the subfigure on the left was obtained by PG-N, the middle one by PG-R and the right one by PG-B. In these figures, the bounding boxes with green border indicate the true positives and the ones with red border signify false positives. Each of the retrieved bounding boxes are also accompanied with a similarity value. The similarity value for each of the bounding boxes is the summation of all the similarities of matched nodes. These similarity values were used to rank the retrievals.

6 Conclusions

In this paper, we have proposed a random walk based approach on TPG to obtain contextual similarities between nodes of the operand graphs of TPG, that capture higher order information. In the next step, these contextual similarities are used to formulate subgraph matching procedure as a node and edge selection procedure in TPG, which is an optimization problem and has been solved with a linear programming relaxation. Since contextual similarities consider neighbour information of nodes/edges, this approach adds more discrimination power to nodes and edges. As a result, graph matching using linear programming turns out to be robust and efficient. We have presented a detailed experimental evaluation with varied difficulty and in most experiments our method performed better than the state-of-the-art methods. In general, it has been revealed that PG-B is more robust than PG-R, which is also supported by the proposal of Aziz et al. [20]. And both PG-B and PG-R perform better than PG-N, which proves the effectiveness of the proposal. Usually, low edge density of the operand graphs results in poor performance, which is well justified since the contextual similarities are generated by the propagation of pairwise similarities through the edges of TPG. So the absence of edges disturbs the similarity values.

Since the optimization problem is solved using linear programming, mostly in inexact cases (e.g. symbol spotting experiment) we do not get a one-one mapping between the nodes and edges. Instead we have performed a simple grouping to cluster the nodes and rank them depending on the total similarity. In future, motivated by Wang et al. [67], this approach can be extended by adding a density maximization based module that would discard the outliers and automatically cluster the inliers.

Bibliography67

The reference list from the paper itself. Each links out to its DOI / PubMed record.

1[1] J. Lladós, E. Martí, J. J. Villanueva, Symbol recognition by error-tolerant subgraph matching between region adjacency graphs, IEEE TPAMI 23 (10) (2001) 1137–1143.
2[2] M. Carcassoni, E. R. Hancock, Spectral correspondence for point pattern matching, PR 36 (1) (2003) 193 – 204.
3[3] J. Hays, M. Leordeanu, A. Efros, Y. Liu, Discovering texture regularity as a higher-order correspondence problem, in: ECCV, Vol. 3952 of LNCS, 2006, pp. 522–535.
4[4] W. Brendel, S. Todorovic, Learning spatiotemporal graphs of human activities, in: ICCV, 2011, pp. 778–785.
5[5] O. Duchenne, A. Joulin, J. Ponce, A graph-matching kernel for object categorization, in: ICCV, 2011, pp. 1792–1799.
6[6] H. Jiang, S. Yu, D. Martin, Linear scale and rotation invariant matching, IEEE TPAMI 33 (7) (2011) 1339–1355.
7[7] H. Li, J. Huang, S. Zhang, X. Huang, Optimal object matching via convexification and composition, in: ICCV, 2011, pp. 33–40.
8[8] A. Dutta, J. Lladós, U. Pal, A symbol spotting approach in graphical documents by hashing serialized graphs, PR 46 (3) (2013) 752–768.