Unsupervised Learning Framework of Interest Point Via Properties Optimization

TL;DR

This paper introduces an unsupervised framework for interest point detection and description using property optimization, employing EM and mini-batch strategies, demonstrating superior performance on matching benchmarks.

Contribution

It proposes a novel unsupervised training framework that jointly learns interest point detectors and descriptors based on property optimization, with an efficient EM-based training strategy.

Findings

Outperforms state-of-the-art methods on image matching benchmarks.

Demonstrates high flexibility across diverse scenes.

Efficient training with Mini-Batch EM reduces computational costs.

Abstract

This paper presents an entirely unsupervised interest point training framework by jointly learning detector and descriptor, which takes an image as input and outputs a probability and a description for every image point. The objective of the training framework is formulated as joint probability distribution of the properties of the extracted points. The essential properties are selected as sparsity, repeatability and discriminability which are formulated by the probabilities. To maximize the objective efficiently, latent variable is introduced to represent the probability of that a point satisfies the required properties. Therefore, original maximization can be optimized with Expectation Maximization algorithm (EM). Considering high computation cost of EM on large scale image set, we implement the optimization process with an efficient strategy as Mini-Batch approximation of EM (MBEM). In the experiments both detector and descriptor are instantiated with fully convolutional network which is named as Property Network (PN). The experiments demonstrate that PN outperforms state-of-the-art methods on a number of image matching benchmarks without need of retraining. PN also reveals that the proposed training framework has high flexibility to adapt to diverse types of scenes.

Tables

Table 1. Table 1: Matching Score of Different Methods

	HP-i	HP-v	W-train	W-test	Oxford
SIFT	0.295	0.314	0.130	0.137	0.357
SURF	0.300	0.281	0.128	0.138	0.395
ORB	0.335	0.306	0.141	0.158	0.421
KAZE	0.362	0.276	0.167	0.182	0.381
LIFT	0.336	0.291	0.172	0.187	0.325
LF-Net	0.299	0.273	0.159	0.175	0.326
Super	0.527	0.458	0.317	0.330	0.434
PN-i-64	0.522	0.472	0.302	0.316	0.445
PN-v-64	0.466	0.503	0.232	0.249	0.521
PN-128	0.469	0.464	0.248	0.259	0.472

Table 2. Table 2: Homography Estimation of Different Methods

	HP-i	HP-v	W-train	W-test	Oxford
SIFT	0.842	0.557	0.526	0.563	0.650
SURF	0.781	0.444	0.396	0.42	0.600
ORB	0.682	0.334	0.304	0.329	0.533
KAZE	0.809	0.398	0.449	0.473	0.492
LIFT	0.875	0.462	0.600	0.628	0.558
LF-Net	0.841	0.435	0.517	0.573	0.567
Super	0.938	0.525	0.713	0.730	0.592
PN-i-64	0.950	0.555	0.802	0.811	0.608
PN-v-64	0.882	0.572	0.564	0.602	0.642
PN-128	0.937	0.554	0.715	0.730	0.617

Table 3. Table 3: Architecture of Property Network

Part	Block	Layer Details
encoder	Conv1	conv, $3\times 3\times 32$, p=1, s=1
		conv, $3\times 3\times 32$, p=1, s=1
		max-pooling, $2\times 2$, strider 2
	Conv2	conv, $3\times 3\times 64$, p=1, s=1
		conv, $3\times 3\times 64$, p=1, s=1
		max-pooling, $2\times 2$, strider 2
	Conv3	conv, $3\times 3\times 64$, p=1, s=1
		conv, $3\times 3\times 64$, p=1, s=1
detection	Conv4-1	trans-conv, $3\times 3\times 64$, p=1, s=2
		concatenate with output of Conv2
		conv, $3\times 3\times 64$, p=1, s=1
	Conv5	trans-conv, $3\times 3\times 64$, p=1, s=2
		concatenate with output of Conv1
		conv, $3\times 3\times 32$, p=1, s=1
		conv, $3\times 3\times 1$, p=1, s=1
		sigmoid function
	NMS	conv, max-pooling, $7\times 7$, strider 1
description	Conv4-2	conv, $3\times 3\times 64$, p=1, s=1
		conv, $3\times 3\times 64$, p=1, s=1
		conv, $3\times 3\times des\_len$, p=1, s=1
	Norm2	$L2$ Normalization for each point
	upsample	upsample with factor=4

Table 4. Table 4: Matching Score of Different Methods on Image Size 320 × 240 320 240 320\times 240

	HP-i	HP-v	W-train	W-test	Oxford
SIFT	0.311	0.299	0.141	0.151	0.405
SURF	0.331	0.289	0.147	0.162	0.44
ORB	0.361	0.284	0.165	0.185	0.447
KAZE	0.398	0.288	0.189	0.205	0.422
LIFT	0.374	0.291	0.202	0.213	0.387
LF-Net	0.372	0.31	0.206	0.219	0.429
Super	0.586	0.462	0.358	0.374	0.496
PN-i-64	0.565	0.466	0.336	0.351	0.483
PN-v-64	0.512	0.513	0.261	0.278	0.572
PN-128	0.516	0.456	0.274	0.288	0.516

Table 5. Table 5: Homography Estimation of Different Methods on Image Size 320 × 240 320 240 320\times 240

	HP-i	HP-v	W-train	W-test	Oxford
SIFT	0.791	0.585	0.453	0.461	0.8
SURF	0.716	0.353	0.314	0.332	0.675
ORB	0.611	0.185	0.237	0.228	0.475
KAZE	0.725	0.336	0.334	0.362	0.592
LIFT	0.826	0.395	0.484	0.517	0.658
LF-Net	0.802	0.384	0.49	0.531	0.675
Super	0.911	0.493	0.664	0.684	0.642
PN-i-64	0.958	0.572	0.764	0.812	0.717
PN-v-64	0.864	0.584	0.495	0.556	0.792
PN-128	0.918	0.588	0.654	0.697	0.775

Equations

$$f_{ij}=F(o_{i},T_{j}(I)\big{|}\theta_{F}),$$

$$IP_{j}=\{o_{i}\big{|}f_{ij}>Pt,o_{i}\in EP\}.$$

$$d_{ij}=D(o_{i},T_{j}(I)\big{|}\theta_{D}),$$

$$DS_{j}=\{d_{ij}\big{|}o_{i}\in IP_{j})\}.$$

$$argma{x}_{{\theta }_F,{\theta }_D}\prod _j\prod _v{P}_v(I{P}_j,D{S}_j).$$

$$s_{loc}(o_{i})=\begin{cases}1,&\left\|\{o_{i^{\prime}}\big{|}o_{i^{\prime}}\in U(o_{i}),o_{i^{\prime}}\in IP_{j}\}\right\|=0\\ 0,&otherwise\end{cases}.$$

$$s_{loc}(I{P}_j)=\prod _{i\in I{P}_j}{s}_{loc}(o_i).$$

$$s_{num}(\Vert I{P}_j\Vert )=\{\begin{array}{ll}1,& N_{min}<\Vert I{P}_j\Vert <N_{max}\\ 0,& otherwise\end{array}.$$

$$s(I{P}_j)=s_{loc}(I{P}_j)\cdot s_{num}(\Vert I{P}_j\Vert ).$$

$$r_i=\underset{J\to \infty }{\lim }\left(\frac{1}{J}\sum _j{f}_{ij}\right).$$

$$r(I{P}_j)=\prod _{o_i\in I{P}_j}{r}_i\prod _{o_i\in EP-I{P}_j}(1-r_i).$$

$$si{m}_{ij{i}^{\prime }{j}^{\prime }}=⟨d_{ij},d_{i^{\prime }{j}^{\prime }}⟩.$$

$$\begin{aligned} c_{i}=&\frac{1}{J(J-1)}\sum_{j}\sum_{j^{\prime}\not=j}\mathbb{I}\left(sim_{ijij^{\prime}}>\right.\\ &\left.max(\{sim_{iji^{\prime}j^{\prime}}\big{|}o_{i^{\prime}}\in IP_{j^{\prime}},i^{\prime}\not=i\}))\right)\\ \end{aligned},$$

$$\widehat{c_i}=exp(\alpha (h_i-H)),$$

$$\begin{aligned} h_{i}=&\frac{1}{J(J-1)}\sum_{j}\sum_{j^{\prime}\not=j}\bigl{(}min(m_{p},sim_{ijij^{\prime}})-\\ &\frac{\lambda}{\left\|IP_{j^{\prime}}-o_{i}\right\|}\sum_{o_{i^{\prime}}\in IP_{j^{\prime}}-o_{i}}max(m_{n},sim_{iji^{\prime}j^{\prime}})\bigr{)}\end{aligned}.$$

$$\hat{c}(I{P}_j,D{S}_j)=\prod _{o_i\in I{P}_j}{\hat{c}}_i.$$

$$argma{x}_{{\theta }_F,{\theta }_D}\prod _k\prod _{j}s(I{P}_j^k)\cdot r(I{P}_j^k)\cdot \hat{c}(I{P}_j^k,D{S}_j^k),$$

$$s_{loc}(y_i^k)=\{\begin{array}{ll}{y}_i^k,& \left({\sum }_{o_{i^{\prime }}\in U(o_i)}{y}_{i^{\prime }}^k\right)=0\\ 0,& otherwise\end{array}.$$

$$s\left(y^k\right)=\mathbb{I}\left(N_{\min }<\Vert y^k\Vert <N_{\max }\right)\cdot \mathbb{I}\left(s_{loc}\left(y^k\right)=y^k\right)$$

$$r\left(y^k\right)=\prod _i{\left(r_i^k\right)}^{y_i^k}{\left(1-r_i^k\right)}^{1-y_i^k}$$

$$h_i^k(y^k)=$$

$$\displaystyle\frac{\lambda}{\left\|y^{k}\right\|}\sum_{i^{\prime}\neq i}{y_{i^{\prime}}^{k}}max\left(m_{n},sim_{iji^{\prime}j^{\prime}}\right)\bigr{)}$$

$$\hat{c}\left(y^k,D{S}_j^k\right)=\prod _i\exp \left(\alpha y_i^k\left(h_i^k\left(y^k\right)-H\right)\right)$$

$$argma{x}_{{\theta }_F,{\theta }_D,y}\prod _{k}s\left(y^k\right)\cdot r\left(y^k\right)\cdot \hat{c}\left(y^k\right).$$

$$\left\{\begin{array}[]{c}{\operatorname{argmax}_{\theta_{F},\theta_{D},y}\prod_{k}r\left(y^{k}\right)\cdot\hat{c}\left(y^{k}\right)}\\ {s.t.N_{\min}<\left\|y^{k}\right\|<N_{\max},s_{loc}\left(y^{k}\right)=y^{k}}\end{array}\right.$$

$$\left\{\begin{array}[]{c}{\operatorname{argmax}_{\theta_{F},\theta_{D},y}L=\sum_{k}\sum_{i}l_{i}^{k}}\\ {s.t.N_{\min}<\left\|y^{k}\right\|<N_{\max},s_{loc}\left(y^{k}\right)=y^{k}}\end{array}\right.$$

$$l_i^k=y_i^k\log r_i^k+\left(1-y_i^k\right)\log \left(1-r_i^k\right)+\alpha y_i^k\left(h_i^k\left(y^k\right)-H\right)$$

$$\left\{\begin{array}[]{c}{P\left(y^{k}|\theta_{F}^{t-1},\theta_{D}^{t-1}\right)=r\left(y^{k}\right)\cdot\hat{c}\left(y^{k}\right)}\\ {s.t.N_{\min}<\left\|y^{k}\right\|<N_{\max},s_{loc}\left(y^{k}\right)=y^{k}}\end{array}\right..$$

$$Y^{k}=\{y^{k}\big{|}N_{min}<\left\|y^{k}\right\|<N_{max},s_{loc}(y^{k})=y^{k}\}$$

$$P\left(y_i^k=1|{\theta }_F^{t-1},{\theta }_D^{t-1}\right)=\frac{1}{Z}\sum _{y^k\in Y^k}{y}_i^k\cdot r\left(y^k\right)\cdot \hat{c}\left(y^k\right),$$

$$E_y(L)=\sum _k\sum _i{E}_{y_i^k}\left(l_i^k\right)$$

Full text

1 Introduction

Interest point is a sparse set of image point containing representative information [3, 18, 22]. With advantage of computation efficiency and representation stability, interest point has a wide utilization in matching tasks such as simultaneous localization and mapping, image registration and stereo reconstruction. Both computation efficiency and representation stability rely on the location and description of interest point which are produced by detector and descriptor respectively.

All interest point related methods (including detectors and descriptors) can be classified as hand-crafted and learning based methods. Hand-crafted methods define some explicit criteria to extract the points [10, 16, 20] and calculate the descriptions while learning based methods are more flexible by training detector and descriptor according to some objectives [23, 24, 9]. Both hand-crafted methods and learning based methods make efforts to lead some important properties to interest point which benefit to diverse applications.

Currently, unsupervised learning is especially feasible to interest point by avoiding the tedious labeling of ground truth points [21, 19]. Following the trend, this paper presents an entirely unsupervised training framework by jointly learning detector and descriptor, which takes an image as input and outputs a probability and a description for every image point. Here the probability reflects how likely the point become interest point, and the description is a feature vector used to represent the unique discriminability of an interest point. In essence, the learning model is to approach to the several optimal properties of interest point. The joint probability distribution of the extracted point properties is used to formulate the objective of our training framework which is maximized to learn detector and descriptor through achieving all desired properties.

The rest of this paper is organized as follows. In section 2 we discuss the related work. In section 3, we formulate our unsupervised learning framework of interest point. Section 4 introduces latent variable to convert the objective function and explains the Expectation Maximization (EM) optimization algorithm. Section 5 instantiate detector and descriptor with fully convolutional network, and show some experiments. Finally, in Section 6 we summarize this paper and list some possible future work.

2 Related Work

The key idea of hand-crafted interest detector is to define explicit criteria to detect and describe interest point. Harris [10], SIFT [16], SURF [4] and KAZE [2] extract the points whose gray value have abrupt changes in two-dimensional space or scale space. FAST [20] detects the points whose local gray distribution conforms to typical corner patterns. Gradient or gradient-like histograms are widely used in float-value description such as SIFT [16], SURF [4], and Harris [2]. Binary descriptors such as BREAF [5], BRISK [13], FREAK [1] select some pixel pairs in neighborhood of interest point to calculate binary feature. Furthermore, scale and orientation estimations [16, 4, 2] are normally integrated into above descriptors to improve matching performance under scale and rotation transformations.

TaSK [23] and TILDE [24] focus on the points extracted by existing detectors (i.e., Forstner [24] and SIFT [16]) and train detectors to improve repeatability of these points under different illuminations. Quad-network [21] approximates repeatability objective with ranking objective and doesn’t rely on any existing detectors. LIFT [26] first selects image patches around SIFT point [16], then train descriptor, orientation estimator and fine-tune detector. With known key point locations, SuperPoint [9] uses synthetic geometry shapes dataset to pre-trains a weak detector which is further trained in active learning way. Descriptor of SuperPoint is learned by another discriminability objective. In LF-Net [19] repeatability objective is approximated by calculating loss of detector between a pair of images, and its discriminability objective impact both descriptions and locations of extracted points, which is similar to LIFT [26]. SIPS [7] fixes descriptor as existing model such as SIFT [16], and train detector to fit the descriptor. IMIP [6] jointly learn detector and descriptor to extract the fixed number of points whose descriptions are limited as one-hot feature vectors.

There are three essential differences between our training framework and existing methods. First, whereas existing methods improve some properties implicitly, we directly maximize the probability that interest point satisfies desired properties. Second, whether a point satisfies the required properties can be thought as latent binary variable so that our training framework can be optimized by Expectation Maximization algorithm. Third, our framework is very flexible to be generalized to any specific application by instantiating different models.

3 Unsupervised Learning Framework of Interest Point

3.1 Problem Formulation

Interest point is a sparse set of image point containing representative and discriminative information. In order to integrate both types of properties, a general framework learns detector and descriptor which extract interest point and its description respectively. First we introduce the general process of extracting interest point from a given scene.

For the sake of describing the problem clearly, we give some notations. A actual world scene to acquire image is denoted as $I$. All possible viewpoint or illumination conditions for taking image are abstracted as transformation set $T=\{T_{j}|j\in Z\}$. Here $T_{j}$ represents a specific condition, and $Z=\{1,2,…,J\}$ represent all possible conditions. The image acquired from $I$ under condition $j$ is expressed as $T_{j}(I)$. Each point in $I$ and its corresponding mapped point in $T_{j}(I)$ are all denoted as $o_{i}$. Suppose the entire scene point set is $EP=\{o_{i}|i=1,2,…,N\}$. Here $N$ represents the number of scene points.

In general detector and descriptor take image $T_{j}(I)$ as input and output interest point and its description respectively. Detector $F$ is defined as a function outputting a probability $f_{ij}$ for every point $o_{i}$ in image $T_{j}(I)$,

[TABLE]

where $\theta_{F}$ are all the parameters of the model of detector, and $f_{ij}$ reflects how likely $o_{i}$ becomes an interest point. In practice probability threshold $Pt$ is introduced to obtain a deterministic interest point set. Interest point set of $T_{j}(I)$ is

[TABLE]

And $EP-IP_{j}$ is named as background point set.

Descriptor $D$ is defined as a function outputting a description vector $d_{ij}$ for every $o_{i}$ in image $T_{j}(I)$, i.e.,

[TABLE]

where $\theta_{D}$ are all the parameters of the model of descriptor, and $d_{ij}$ can be used to calculate the similarity between this point and other interest point, which is very important to determine the discriminability of an interest point. We always ensure $\left\|d_{ij}\right\|_{2}=1$ with length normalization. The description set of image $T_{j}(I)$ is denoted as

[TABLE]

The problem of learning based interest point is equivalent to learning the parameters of detector and descriptor. Whereas there is no unique supervised label for interest point and its description, it’s reliable to jointly train detector and descriptor in an unsupervised way which focuses on the essential properties of interest point. Therefore, we propose an unsupervised training framework to optimize the learning model of interest point.

The overview of the framework is shown as Figure 5. With images acquired from the same scene, this framework jointly trains detector and descriptor. The training samples can be transformed from a scene image by simulating different illumination and viewpoint changes. Images are fed into detector and descriptor which outputs a probability and a description for every image point. Then, the joint probability of interest point properties is computed through the above two produced information. Finally, a proposed expectation maximization algorithm optimizes the joint probability to find the best model parameters, which means that the outputs of detector and descriptor have achieved all desired properties.

3.2 Unsupervised Properties Optimization

The probability that interest point of $T_{j}(I)$ satisfies given the $v$th property can be formulated as $P_{v}(IP_{j},DS_{j})$. Here $v\in\{1,2,…,V\}$ and $V$ is the number of all desired properties. Suppose all properties are independent and properties in different images are also independent. The objective maximizing the probability that interest point satisfies desired properties is

[TABLE]

In this paper we make $V=3$, and sparsity, repeatability and discriminability are selected as essential properties in this paper. The probabilities that an interest point satisfies above properties are denote as sparsity probability, repeatability probability and discriminability probability, whose formulations are introduced in subsequent subsections.

3.3 Sparsity Probability

Sparsity is an essential property controlling the limited number of interest points sparsely scatter over image. According to definition sparsity only rely on interest point rather than its description, so its probability can be represented as $s(IP_{j})$.

As a specific implementation, we define $s_{loc}(IP_{j})$ as the probability that interest point set of image $T_{j}(I)$ is locally sparse. Local sparsity indicates that there is none in the interest point $o_{i}$’s neighborhood except itself. $U(o_{i})$ represents the neighborhood of point $o_{i}$ whose radius is $rad$. Here $rad$ is a small integral (e.g., $rad=4$), and $U(o_{i})$ doesn’t contain $o_{i}$ itself. Define $\left\|\cdot\right\|$ returns the number of elements in a given set. Then the local sparsity probability of $o_{i}$ is defined as

[TABLE]

Suppose the local sparsity probabilities of different interest points is independent, the local sparsity probability of entire interest point $IP_{j}$ is

[TABLE]

Except making interest point to be sparse locally, sparsity is also a global properties controlling the number of interest points. Depending on the number of interest points $\left\|IP_{j}\right\|$, $s_{num}(\left\|IP_{j}\right\|)$ represents the probability that number of interest point is reasonable. We define it as

[TABLE]

Combining Equation 7 and 8, the sparsity probability is expressed as

[TABLE]

3.4 Repeatability Probability

Repeatability indicates how likely a point can be extracted repeatedly under different viewpoints and illuminations. With this definition repeatability is determined on multiple images acquired from the same scene rather than a single image, so repeatability is defined on $I$ to simplify notations. Denote repeatability probability of point $o_{i}$ in $I$ as $r_{i}$, which represents the probability that $o_{i}$ is extracted in $I$, i.e.

[TABLE]

where $f_{ij}$ is the probability output by detector for $o_{i}$ in image $T_{j}(I)$ which is defined in Equation 1. Then $1-r_{i}$ is the probability point $o_{i}$ belongs to background point set. In the reminder of the paper we ignore the limit of $J$ to simplify notations. Suppose the repeatability of each point is independent on each other, the repeatability probability of $IP_{j}$ satisfies

[TABLE]

3.5 Discriminability Probability

Discriminability denotes how likely an interest point in one image is more similar to the same point than the other interest points in another image. The similarity between point $o_{i}$ and $o_{i^{\prime}}$ is normally defined as the inner product of their description vector $d_{ij}$ and $d_{i^{\prime}j^{\prime}}$, which is formulated as

[TABLE]

$sim_{iji^{\prime}j^{\prime}}\in[-1,1]$ because $\left\|d_{ij}\right\|_{2}=1$. If $i=i^{\prime}$, $sim_{iji^{\prime}j^{\prime}}$ denotes the similarity between the same point in two images, which is termed as positive pair. Otherwise when $i\not=i^{\prime}$, $sim_{iji^{\prime}j^{\prime}}$ represents the similarity between different points in two images, which is termed as negative pair.

Denote indicator function as $\mathbb{I}(\cdot)$ which return 1 if and only if logical operation is true. Define function $max$ returning the maximum of a set and returning $–Inf$ for empty set. Then the discriminability probability of an interest point is

[TABLE]

where $o_{i^{\prime}}$ must be an interest point rather than an arbitrary point because only interest point will be considered in matching process. In practice Equation 13 is too sharp to represent the gap between current and optimal discriminability. So we approximate it with

[TABLE]

where $h_{i}$ is the difference between similarity of positive pair and negative pairs, and $H$ is the maximum of $h_{i}$. $\alpha$ is a factor controlling the sensitivity of discriminability probability $\hat{c}_{i}$ with respect to $h_{i}$. The formulation of $h_{i}$ is

[TABLE]

The formulation of $h_{i}$ is inspired by descriptor loss in [7]. Here $m_{p}\in[-1,1]$ and $m_{n}\in[-1,1]$ are named as positive margin and negative margin, which can be seemed as the target similarity of positive pair and negative pair. $\lambda$ is a weight to balance positive pair and negative pair. Because $sim_{iji^{\prime}j^{\prime}}\in[-1,1]$ the maximum of $h_{i}$ is $H=m_{p}-\lambda m_{n}$.

Suppose the discriminability of each point is independent, the discriminability probability of $IP_{j}$ is

[TABLE]

Note in Equation 16 we don’t concern discriminability between interest point and background point, or discriminability between background points themselves.

3.6 Objective of Properties Optimization

During training, we generally have a scene set $\{I^{k}|k=1,2,...,K\}$. Denote interest point set of image $T_{j}(I^{k})$ as $IP_{j}^{k}$ where the descriptor of each point $o_{i}$ is $d_{ij}^{k}$. Thus, description set of image $T_{j}(I^{k})$ is $DS_{kj}=(d_{ij}^{k}|o_{i}\in IP_{kj})$. Notations $r_{i}$, $h_{i}$, $\hat{c}_{i}$ are also extended as $r_{i}^{k}$, $h_{i}^{k}$, $\hat{c}_{i}^{k}$ to represent repeatability and discriminability for point $o_{i}$ in scene $I^{k}$. Then the objective of properties optimization is

[TABLE]

which is named as properties objective. Note the description set $DS_{j}^{k}$ is ignored in $s$ and $r$ because sparsity and repeatability probability don’t concern description of interest point.

4 Optimization and Implementation

4.1 Problem Conversion with Latent Variable

It’s hard to straightforward optimize properties objective with conventional gradient based algorithm. In Equation 2 interest point set $IP_{j}$ is determined by the probability threshold $Pt$ where the derivative of the logical operation doesn’t exist. Therefore, we introduce latent variable to solve this problem.

Binary latent variable $y_{i}^{k}$ is formulated for every point $o_{i}^{k}$ in scene $I^{k}$ and $y_{i}^{k}=1$ if and only if point $o_{i}$ in any image $T_{j}(I^{k})$ satisfy all desired properties. $y^{k}$ is the vector whose $i$th component is $y_{i}^{k}$, and $Y$ is the matrix whose item in $k$th row and $i$th col is $y_{i}^{k}$. Point $o_{i}^{k}$ is defined as satisfied point if $y_{i}^{k}=1$, and satisfied point set of $I^{k}$ is $\{o_{i}|y_{i}^{k}=1\}$.

In original objective 17 $\theta_{F}$ and $\theta_{D}$ are optimized to make interest point sets $IP_{j}^{k}$ and their descriptions achieve the desired properties, meaning that optimal solution of $IP_{j}^{k}$ must be the point set $\{o_{i}^{k}|y_{i}^{k}=1\}$. So replace $IP_{j}^{k}$ as $\{o_{i}^{k}|y_{i}^{k}=1\}$ in Equation 17 won’t change its optimal solution. We first convert the property probability to more compact formulations with $y_{i}^{k}$. The number of satisfied points can be calculated with $\left\|y^{k}\right\|=\sum_{i}y_{i}^{k}$ . Redefine local sparsity probability for $y_{i}^{k}$ as

[TABLE]

For vector $y^{k}$ we define the result of $s_{loc}(y^{k})$ is still a vector, whose $i$th component is $s_{loc}(y_{i}^{k})$.

Redefine sparsity probability of $y^{k}$ as

[TABLE]

The conversion of repeatability probability is straightforward.

[TABLE]

Also we can redefine discriminability probability as

[TABLE]

[TABLE]

Note $\hat{c}(y^{k},DS_{j}^{k})$ for any $j$ is same because all images of $I^{k}$ share the same $y^{k}$, so we replace it with $\hat{c}(y^{k})$ in this case. In fact, all property probabilities for different images of $I^{k}$ are exactly equal by sharing $y^{k}$. Then properties objective 17 can be converted as latent properties objective

[TABLE]

According to Equation 19 sparsity probability $s(y^{k})$ is either 1 or 0. To maximize latent properties objective $s(y^{k})$ has to be ensured as 1, so that objective 23 is equivalent to

[TABLE]

With logarithm transformation it’s equivalent to

[TABLE]

where log-likelihood item

[TABLE]

In objective 25 $L$ is named as log-likelihood function.

4.2 Expectation Maximization of latent properties objective

Objective 25 can be optimized with Expectation Maximization algorithm (EM). Suppose the optimization contains $T$ times iterations. Let $\theta_{F}^{0}$ and $\theta_{D}^{0}$ are initial parameters, and the parameters of detector and descriptor are $\theta_{F}^{t}$ and $\theta_{D}^{t}$ after $t$ times iterations.

E-step:

In E-step of iteration $t$, we need to obtain the expectation of log-likelihood function $L$ with respect to $y$, which is denoted as $E_{y}(L)$. To achieve it we first estimate the probability distribution $P(y|\theta_{F}^{t-1},\theta_{D}^{t-1})$. Because different $y^{k}$ are independent we only discuss the distribution of $y^{k}$ directly. The formulation for probability of $y^{k}$ is

[TABLE]

Denote all possible $y^{k}$ form the set

[TABLE]

Then the probability distribution of $y_{i}^{k}$ is

[TABLE]

where $Z=\sum_{y^{k}\in Y^{k}}r(y^{k})\cdot\hat{c}(y^{k})$ is the normalization factor. Denote $p_{i}^{k}=P(y_{i}^{k}=1)$, then the expectation of $y_{i}^{k}$ is $p_{i}^{k}$. So the expectation of $L$ is

[TABLE]

Here

[TABLE]

M-step:

In M-step $\theta_{F}^{t}$ and $\theta_{D}^{t}$ are obtained by maximizing the expectation of $L$. Normally the number of parameters is very huge, so Gradient Ascent algorithm (GA) is selected to achieve M-step in this paper. The gradient is computed as:

[TABLE]

[TABLE]

Similarity $sim_{iji^{\prime}j^{\prime}}$ is computed with inner product which is derivable. So if the detector $F$ and descriptor $D$ are derivable, their parameters $\theta_{F}$ and $\theta_{D}$ can be optimized with GA. Theoretically, from $\theta_{F}^{t}$ and $\theta_{D}^{t}$ to $\theta_{F}^{t+1}$ and $\theta_{D}^{t+1}$ GA may need multiple times updates to achieve convergence. In practice, both Equation 29 and 44 lead to very high computational complexity, so we introduce Mini-Batch Approximation as an efficient implementation. All details of of this approximation are outlined in Supplementary Section 1.

5 Experimental Result

5.1 Experiment Setup

Our framework is flexible to be integrated with different models. In this paper we implement detector $F$ and descriptor $D$ as two Fully Convolutional Network [15, 27] with Batch Normalization [11], and parameters of their encoders are shared. We name this implementation as Property Network (PN). The architecture of PN is demonstrated in Supplementary Section 2.

In this paper we select MS-COCO 2014 [14] as training dataset, which comprises of more than 80 thousand images. We treat each single image as a scene which is transformed to the training images with different illuminations and viewpoints through simulated transformations. The details of simulated transformations are outlined in Supplementary Section 3.

To adapt to different kinds of illumination and viewpoints changes, we train three different PN models corresponding to three transform simulation conditions. PN-i-64 model is trained with the condition of sharp illumination change and medium viewpoint change. PN-v-64 is trained with the condition of large range viewpoint change and medium illumination change. PN-128 is trained with the condition of sharp illumination change and large range viewpoint change. The architectures of above three networks are identical except the length of their description vector. The length of description vector of PN-i-64, PN-v-64 and PN-128 are 64, 64, 128 respectively.

The configurations of all hyperparameters of PN are as below. During training we resize all images to $320\times 240$ and stack different images to mini-batches (but the size of testing image can be arbitrary). In every iteration two scenes are randomly transformed ten times respectively, so there are in fact twenty images in a mini-batch (i.e., $B=2$ and $J=10$). In entire training we fix number range $N_{min}=200$, $N_{max}=400$, and non-maximum suppression radius $rad=4$ pixels. For discriminability, negative pair weight $\lambda=10/N_{max}$ , positive margin $m_{p}=1$, negative margin $m_{n}=0.2$ and discriminability weight $\alpha=1$. We always stop training after two epochs. The FCN model is implemented with and solved by Adam optimizer [12] with default parameters ($lr=0.001$ and $\beta=(0.9,0.999)$).

5.2 Performance Comparison

Three datasets HPatches [3], Webcam [24], Oxford [17] are used to evaluate the performance. HPatches is divided into illumination changed subset and viewpoint changed subset, which are denoted as HP-i and HP-v respectively. Webcam contains sharp illumination changes but no viewpoint change. In this paper no method is trained on Webcam, so training set and testing set of Webcam are both available to evaluate performance, which is denoted as W-train and W-test. Oxford comprises of both illumination and viewpoint changes, but we don’t split it because the number of images in Oxford is small.

Two used metrics Matching Score and Homography Estimation are identical to that used in [9] (see more details in Supplementary Section 4). Higher Matching Score and Homography Estimation are better. Briefly, Matching Score measures the ratio of the recovered ground truth correspondences over the number of points extracted by the detector in the shared viewpoint region. Homography Estimation measures the ability of an algorithm to estimate the homography that relates a pair of images by comparing it to ground truth homography. Similar to [9], we fix correct distance $\epsilon=3$ pixels, and use RANSAC method implemented by Opencv Toolbox to estimate homography. In this paper all evaluations are performed on image size $640\times 480$. To be fair we use the recommended hyperparameters for every method, except the maximum number of extracted interest points. We keep no more than 1000 interest points for $640\times 480$ images.

The performances of different methods are shown in Table LABEL:tab:tabscore and Table LABEL:tab:tabhomo.

In summary, our Property Networks outperform the other methods on viewpoint changed subset of HPatches and Oxford, and achieve competitive performance on others according to Matching Score. With Homography Estimation, our methods outperform other methods on HPatches and Webcam and achieve competitive performance on Oxford. As the state-of-the-art algorithms, SIFT has low performance on illumination changed datasets and SuperPoint has no superiority on viewpoint changed datasets, but our Property Networks present much more stable results on all above datasets. Furthermore, it’s interesting that SuperPoint has slight advantage on Webcam according to Matching Score but has large gap comparing to our PN-i-64 according to Homography Estimation. This is because Matching Score is a basic metric only reflecting the maximum number of points which are likely helpful to subsequent tasks. However, the space distribution of matched points is as important as the number of matched points for the final tasks such as homography estimation and camera pose estimation, which can’t be revealed by Matching Score. Sparsity is an essential property making the space distribution of interest point suit for these tasks. By improving sparsity and other properties jointly PN-i-64 achieve much higher performance for Homography Estimation on illumination changed dataset.

Figure 5 demonstrates some visualization details about it. Whereas SuperPoint and SIFT achieve the best performance in some specific scenes, PN-i-64 model give reliable results under both illumination and viewpoint changes. Furthermore, above visualization results of PN-i-64 and SuperPoint explains the difference between Matching Score and Homography Estimation. More results can be found in Supplementary Section 5.

5.3 Performance Analysis of Three PM Models

It’s an ultimate goal to learning an interest point model that can be generalized to all application scenes. From the viewpoint of actual application, it’s practical to learning an interest model to cope with specific conditions. The results of our three models confirm this opinion, which is intuitively demonstrated in Figure 6.

Though PN-128 has the highest potentials with largest description vector length, in our training it can’t converge well facing both sharp illumination changes and large range viewpoint changes, which makes it have no superiority for either illumination or viewpoint changes. From another perspective, if there are both sharp illumination changes and large range viewpoint changes in given application, PN-128 should be a reasonable tradeoff with limited learning ability. Note PN-128 achieves similar or better results comparing with SuperPoint whose length of description vector is 256.

By focusing on sharp illumination changes and medium viewpoint changes, PN-i-64 achieves much better performance on illumination changed datasets, which demonstrates the flexibility of our training framework. What we need is only feeding the model corresponding images for different application scenes, without adjusting the architecture and objective anymore. PN-v-64 outperform PN-128 under viewpoint changes, but the superiority is relatively small. One reason is conventional convolution neural network such as Fully Convolutional Network can only achieve limited rotation invariant with convolution and pooling operation [8, 25]. Explicitly estimating the orientation of interest point is a one of reliable ways to improve rotation invariant [16, 26].

6 Conclusion

This paper proposes an entirely unsupervised training framework by maximizing the sparsity, repeatability and discriminability probability of interest point and its description. With Expectation Maximization algorithm and mini-batch approximation this framework can be optimized efficiently. As an implementation based on Fully Convolutional Network, Property Network outperforms state-of-the-art algorithms on a number of image matching dataset, which demonstrates the effectiveness and flexibility of our training framework. Future work will investigate more well-designed architecture of detector and descriptor which can reach more potential of this framework. Furthermore, our framework can be integrated with more properties to improve model performance in diverse applications, and some supervised information can also be formulated as properties which bring semantics to interest point.

Supplementary Material

In this supplementary material we give more details of our implementation and experimential results. In section 1 we implement the Expectation Maximization (EM) process with an efficient strategy called Mini-Batch approximation of EM (MBEM). Section 2 demonstrates the architecture of our Property Network. Section 3 outlines details of our simulation for illumination and viewpoint changes. In Section 4 we introduce performance metrics used in experiments and more experimential results are demonstrated in Section 5.

1 Mini-Batch Approximation of Expectation Maxmization

1.1 Problems of Original Expectation Maxmization

Before discuss the difficultes of optimizing our objective with original Expectation Maxmization algorithm (EM), we first overview the objective of our training framework, which is formulated as

[TABLE]

where $\theta_{F}$ and $\theta_{D}$ are parameters of detector and descriptor respectively, and $y^{1},...,y^{K}$ are latent variables. Each $y^{k}$ is correspond to a scene $I^{k}$, and $y^{k}$ is a vector whose component $y_{i}^{k}$ is a binary random variable representing whether point $o_{i}^{k}$ satisfies desired properties. Point $o_{i}^{k}$ is defined as satisfied point if $y_{i}^{k}=1$. Note $y^{k}$ need to be optimized because we don’t known which points can become satisfied points. To be convenient for optimization we also define log-likelihood function

[TABLE]

Because each $y^{k}$ is independent in objective (34) , we drop the superscript in the following parts of the supplement for the brief expression. Without ambiguity we always use $L$ to represent log-likelihood function no matter whether the $k$ is ignored or not.

Properties considered in objective (34) are sparsity, repeatability and discriminability. The repeatability and discriminability probability is

[TABLE]

[TABLE]

where $r_{i}$ and $\hat{c}_{i}(y)$ are probabilities that $o_{i}$ satisfies repeatability and discriminability respectively. The formulations of $r_{i}$ and $\hat{c}_{i}(y)$ can be found in main text. In this section we only need know $r_{i}$ don’t rely on $y$, but $\hat{c}_{i}(y)$ depend on vector $y$.

The constraint in objective (34) is named as sparsity constraint. $\left\|y\right\|=\sum_{i}{y_{i}}$ represents the number of satisfied points, and $[N_{min},N_{max}]$ are reasonable number range of interest points in a single scene. Local sparsity constraint $s_{loc}$ is defined as

[TABLE]

where $U(o_{i})$ represents the neighborhood of point $o_{i}$ whose radius is $rad$. Here $rad$ is a small integral (e.g., $rad=4$), and $U(o_{i})$ doesn’t contain $o_{i}$ itself.

Theoretically, objective (34) can be optimized with EM. Normally EM contains $T$ iterations, and an Expectation step (E-step) and a Maximization step (M-step) are conducted in each iteration. Let $\theta_{F}^{0}$ and $\theta_{D}^{0}$ are initial parameters, and the parameters of detector and descriptor are $\theta_{F}^{t}$ and $\theta_{D}^{t}$ after $t$ iterations. In each E-step we need obtain $p_{i}=P\left(y_{i}=1|\theta_{F}^{t-1},\theta_{D}^{t-1}\right)$ and compute the expectation of $L$ with respect to $y$ which is denoted as $E_{y}(L)$. In each M-step we need update $\theta_{F}^{t-1}$ and $\theta_{D}^{t-1}$ to $\theta_{F}^{t}$ and $\theta_{D}^{t}$ by maximizing $E_{y}(L)$.

We first formulate $p_{i}$ theoretically. Without sparsity constraint the sample space of $y$ can be represented as $\{y^{(m)}|m=1,2,...,2^{N}\}$ where $N$ is the number of all the points in given scene. That’s because the length of vector $y$ is $N$ and each component of $y$ is a binary variable. But with sparsity constraint this sample space should be more narrow. Denote the sample space of $y$ as $Y$ which can be formulated as

[TABLE]

The sparsity constraint in objective (34) can be ignored if we ensure $y\in Y$. So when $y\in Y$ the distribution of $y$ can be reformulate as

[TABLE]

Note we obtain (40) by substituting (36) and (37) into (34).

Because the distribution $p_{i}$ is a marginal distribution of $P(y|\theta_{F}^{t-1},\theta_{D}^{t-1})$, so we obtain

[TABLE]

where $Z=\sum_{y^{(m)}\in Y}r(y^{(m)})\cdot\hat{c}(y^{(m)})$ is the normalization factor, and $Y_{y_{i}=1}=\{y^{(m)}|y^{(m)}_{i}=1,y^{(m)}\in Y\}$. By combining 35, 36 and 37, the expectation of $L$ with respect to $y$ is

[TABLE]

Note we drop all the $k$ of $L$ and only considering single $y$. Here $\hat{c}_{i}(y)$ depend on $y$ so we can’t straightforward simplify the expectation $E_{y}(y_{i}\log(\hat{c}_{i}(y)))$ in (42).

The $E_{y}(L)$ need be maximized in M-step, which can be achieved with Gradient Ascent algorithm (GA). According to definitions in main text, $r_{i}$ depends on $\theta_{F}$, and $\hat{c}_{i}(y)$ depends on $\theta_{D}$. The expressions of partial derivative are

[TABLE]

[TABLE]

Here $\frac{\partial r_{i}}{\partial\theta_{F}}$ in (43) can be computed directly, but the partial derivative in (44) is more complex.

Above E-step and M-step lead to high computational complexity from three aspects. 1) It’s inefficient to make GA achieve convergence on entire image set in each M-step (note (43) and (44) are defined for one scene but in fact we need compute them for all scenes). 2) Directly computing $p_{i}$ with (41) need traverse $y^{(m)}$ through entire $Y$, which leads to very high computational complexity. 3) Even with known $p_{i}$, computing (44) still need traverse $y^{(m)}$ through entire $Y$.

In the subsequent subsection, we solve above three problems with some approximations of original EM. The entire approximate algorithm is named as Mini-Batch approximation of Expectation Maxmization (MBEM).

1.2 Considering Mini-Batch in Each Iteration

Because it’s too slow to make GA achieve convergence on entire image set in each M-step, in each iteration we only focus on a small subset of training set, which is normally named as mini-batch.

Here is more details. First we select size of mini-batch as $BS$ (in our training we fix $BS=2$). Construct $B_{t}=\{b|t\cdot BS+1\leq b\leq(t+1)BS\}$ in iteration $t$ and select mini-batch $\{I^{k}|k\in B_{t}\}$ to conduct E-step and M-step. Furthermore, each M-step only updates parameters once rather than multiple times until converge, because it’s not necessary to achieve convergence on a mini-batch which increases the risk of overfitting.

1.3 Efficient Approximation for Distribution of Latent Variable

Though distribution $p_{i}$ can be computed with (41), but this computation need traverse $y^{(m)}$ through entire sample space $Y$ which leads to very high computational complexity. There are two obstacles to simplify (41).

The definition of sample space $Y$ is integrated with sparsity constraint. It’s easy to check whether a specific $y^{(m)}$ satisfies sparsity constraint or not, but it’s difficult to straightforward obtain all satisfied $y^{(m)}$. 2. 2)

To solve distribution $p_{i}$, we must consider all points because $\hat{c}_{i}$ depends on vector $y$.

Whereas it’s hard to solve (41) precisely, this subsection introduce an efficient strategy to approximate $p_{i}$ which comprises of three step. First, we approximate sample space $Y$ with $\hat{Y}$ which can be obtained efficiently. Second, the $\hat{c}_{i}(y)$ is approximated with $\hat{c}_{i}(\hat{y})$ where $\hat{y}$ is a constant for given $\theta_{F}$. Third, we simplify the approximate formulation of $p_{i}$.

1.3.1 Efficient Approximation of Sample Space

We first approximate $Y$ with $\hat{Y}$ to simplify the sparsity constraint. A reasonable $\hat{Y}$ should have two characteristics. First, all $y\in\hat{Y}$ satisfy sparsity constraint. That’s mean $\hat{Y}$ can only be a subset of $Y$, i.e., we can only remove some elements from $Y$ in order to obtain $\hat{Y}$. Second, replace $Y$ with $\hat{Y}$ won’t change $p_{i}$ significantly. That’s mean elements in $\hat{Y}$ should have greater contributions to $p_{i}$ comparing to elements in $Y-\hat{Y}$.

The first characteristic can be achieved by setting $y_{i^{\prime}}=0$ for all $o_{i^{\prime}}\in U(o_{i})$ if we want to setting $y_{i}=1$. The remaining question is how to select point $o_{i}$ to set $y_{i}=1$. According to (41), the larger $r_{i}$ and $\hat{c}_{i}(y)$ point $o_{i}$ has, the greater contribution $y_{i}=1$ can make. Considering $r_{i}$ can be obtained directly but $\hat{c}_{i}(y)$ depend on $y$, and we have assume $r_{i}$ and $\hat{c}_{i}(y)$ are independent in main text, so we can select point $o_{i}$ if $r_{i}$ is a local maximum and set $y_{i}=1$. According to above analyses we define

[TABLE]

where $max(\cdot)$ return the maximum of given set. Define $\hat{y}$ is a vector whose $i$th component is $\hat{y}_{i}$. Since the local maximum points that are deducted from the current model require the corresponding points in the sample $y^{m}$ must be also local maximum as $\hat{y}$, the sample space Y can be further reduced as

[TABLE]

Note any $y\in\hat{Y}$ satifies $s_{loc}(y)=y$ becuase $s_{loc}(\hat{y})=\hat{y}$, this significantly benefit computation efficiency by avoid checking this constraint.

1.3.2 Efficient Approximation of $\hat{c}_{i}(y)$

The second problem is $\hat{c}_{i}$ depends on vector $y$ which make us have to consider all points when solve $p_{i}$. To reduce the computational complexity, we give an approximation of $\hat{c}_{i}(y)$ with $\hat{c}_{i}(\hat{y})$. We first make an assumption called

Discriminability Consistency:

Given two feasible samples $y^{(1)},y^{(2)}\in\hat{Y}$ which satisfy $\{o_{i}|y^{(1)}_{i}=1\}\subseteq\{o_{i}|y^{(2)}_{i}=1\}$, discriminability consistency assumes $\forall i,i^{\prime}\in\{o_{i}|y^{(1)}_{i}=1\}$, if $\hat{c}_{i}(y^{(1)})<\hat{c}_{i^{\prime}}(y^{(1)})$, then $\hat{c}_{i}(y^{(2)})<\hat{c}_{i^{\prime}}(y^{(2)})$.

Discriminability consistency assumes if the discriminability of point $o_{i}$ is larger than $o_{i^{\prime}}$ in satisfied point set $\{o_{i}|y_{i}=1\}$, then inserting some interest points into this satisfied point set may not change their relative order. According to the definition $\hat{y}$ which is deduced from the local maximum $r$ computed with current model parameters, $\hat{y}$ must be the super set of all the other genuine interest point sets.

If we view any sample $y$ as $y^{(1)}$ and $\hat{y}$ as $y^{(2)}$, then the discriminability consistency will be satisfied. Our objective of optimization is to maintain the best descriminability of each interest point so that we can use $\hat{y}_{i}$ to replace $y_{i}$. Therefore,it’s reasonable to select $\hat{y}$ to replace any sample $y$ to guarantee discriminability consistency. Furthermore, the feasible samples are also replaced by $\hat{y}$ such that we only concern about the computation of $\hat{y}$. With $\hat{y}$ we can obtain

[TABLE]

where $h_{i}$ is the function computing disciminability for given point $o_{i}$, and its formulation can be found in main text.

1.3.3 Simplify the Formulation of $p_{i}$

First we give the formulation of $p_{i}$ combining above two approximations. Because $\hat{y}_{i}$ is a constant with given $r_{i}$, we denote $\hat{c}_{i}(\hat{y})$ as $\tilde{c}_{i}$ for the simplicity. Then (41) can be approximated with

[TABLE]

Here $Z$ is still the normalization factor and we don’t show its formulation repeatedly, and $Y_{y_{i}=1}$ in (41) is changed to $\hat{Y}_{y_{i}=1}=\{y^{(m)}|y_{i}=1,y^{(m)}\in\hat{Y}\}$.

In order to ensure computation efficieny we need avoid the summation over $y^{(m)}$, the product over $i^{\prime}$ and the computation of normalization factor $Z$ in (48). In this subsection we intorduce our solution to achieve it.

We make $z_{i}^{(m)}={r_{i}^{y_{i}^{(m)}}\cdot(1-r_{i})^{1-y_{i}^{(m)}}\cdot\tilde{c}_{i}^{y_{i}^{(m)}}}$ for the simplicity , then (48) can be reformulated as

[TABLE]

Then we formulate $1-p_{i}$ according to (40), i.e.,

[TABLE]

where $\hat{Y}_{y_{i}=0}=\{y^{(m)}|y_{i}=0,y^{(m)}\in\hat{Y}\}$.

Make $\|\cdot\|$ return the number of elements in given set, then (49) and (50) can be converted to

[TABLE]

[TABLE]

where

[TABLE]

[TABLE]

Whether $y_{i}=1$ or $y_{i}=0$ only slightly change $\prod_{i^{\prime}\neq i}{z_{i^{\prime}}}$, and this difference can be ignored after average them over $\hat{Y}_{y_{i}=1}$ and $\hat{Y}_{y_{i}=0}$, so $avg\left(\hat{Y}_{y_{i}=1}\right)\approx avg\left(\hat{Y}_{y_{i}=0}\right)$. With (51) and (52) we can obtain

[TABLE]

Note $r_{i}$ and $\tilde{c}_{i}$ can be computed efficiently without interaction to $\hat{Y}$. So only $\|\hat{Y}_{y_{i}=1}\|$ and $\|\hat{Y}_{y_{i}=0}\|$ need to be determined in (55).

We first compute $\|\hat{Y}\|$ with (46). Each $y$ in $\hat{Y}$ corresponds to a set of satisfied points which is select from $\{o_{i}|\hat{y}_{i}=1\}$. This selection can be splited into two steps. First, we determine the number of satisfied points, i.e., select $n\in[N_{min},N_{max}]$. Second, we choose $n$ point from $\{o_{i}|\hat{y}_{i}=1\}$, and the corresponding $y$ is obtained. According to above two step $\|\hat{Y}\|$ can be calculated with

[TABLE]

where $C_{\|\hat{y}\|}^{n}$ indicate the number of combinations.

In computation of $\|\hat{Y}_{y_{i}=1}\|$, we can only choose $\|y\|-1$ satisfied points because we have select $o_{i}$ as a satisfied point. So

[TABLE]

Then

[TABLE]

Substituting (57), (58) and definitions of $r_{i}$ and $c_{i}$ (details can be found in main text) into (55), the approximate $p_{i}$ can be computed efficiently.

1.4 Efficient Approximation for Partial Derivative of Description Parameters

In Subsection 1.3 we can obtain distribution $p_{i}$ efficiently. But even with known $p_{i}$ computing $E_{y}(y_{i}\log(\hat{c}_{i}(y))$ and its partial derivative in (44) still need traverse $y$ through entire $Y$ because $\hat{c}_{i}$ depends on $y$. Fortunately, in Subsection 1.3 we have solve this problem by approximating $y$ with $\hat{y}$ under the assumption of discriminability consistency. Because $\hat{y}$ is independent of $y$, so

[TABLE]

where $\tilde{c}_{i}=\hat{c}_{i}(\hat{y})$ is defined in Subsection 1.3. Substituting (59) into (44) we obtain

[TABLE]

According to the definition of discriminability in main text, $\frac{\partial\tilde{c}_{i}}{\partial\theta_{D}^{t-1}}$ can be computed directly. So the entire Gradient Ascent algorithm can be performed.

2 Architecture of Property Network

Property Network (PN) is a specific implementation of our training framework. In PN, both detector and descriptor are implemented with Fully Convolutional Network [15, 27], whose architecture is inspired by [9]. Figure 5 shows the architecture of PN briefly. Detector comprises of encoder and detection decoder, and descriptor comprises of encoder and description decoder. Both detector and descriptor share the same encoder. Table LABEL:tab:arch denomstrate more details of architecture of PN.

In Table LABEL:tab:arch, We ignore all Batch Normalization and ReLU layers, which follow every convolution layer except the final convolution layer in Conv5 and Conv4-2. “Concatenate” operation concatenates current feature map and given feature map into a new feature map. ”trans-conv” indicate transposed convolution [27]. p=1 means padding is 1, and s=1 indicates the strider of sliding window is 1. $des\_len$ is the length of description vector. As introduced in main text, we make $des\_len=64$ in PN-i-64 and PN-v-64, and $des\_len=128$ in PN-128.

3 Simulations for Viewpoint and Illumination Changes

In this section we outline our simulations for illumination and viewpoint changes. For illumination simulation we randomly select the transformations to change pixel value. For viewpoint simulation we randomly generate homography matrices used to perform homography transformation. Figure 5 shows several examples of our simulated results. We first introduce more details of simulations for illunimation changes. Our slmulations comprise of seven kinds of transformations. In the discussion below we assume the original image have three color channels with 256 gray levels.

Image Blur. Apply Gaussian blur, average blur or median bluron image. 2. 2)

Channels Shuffle. Permute the order of the color channels of image. 3. 3)

Contrast Normalization. Change the contrast in images by moving pixel values away or closer to 128. 4. 4)

Grayscale. Convert images to grayscale and mixe with the original image with a random weight. 5. 5)

Invert all pixels in given image, i.e. set them to $255-original\_pixel\_value$. 6. 6)

Salt and Pepper noise. Randomly replace some pixels with very white or black colors. 7. 7)

Shadow. Randomly insert some dark shapes into image.

In main text we have mentioned different PN models are trained with different level of simulations. PN-v-64 is trained with only Image Blur, Contrast Normalization and Shadow, and PN-i-64 and PN-128 are trained with all kind of simulations for illumination changes.

In simulations for viewpoint changes, we randomly generate homography matrices and use them to perform homography transformation. Different PN models are also trained with different level of homography transformation. In training of PN-i-64, we ensure the rotation angle of homography transformation to be less than $45^{\circ}$, and for PN-v-64 and PN-128 we don’t conduct restriction of rotation angle.

All above simulations are online, i.e., in each training iteration we randomly perform above transformations on current image mini-bacth, and all transformed images will be discarded after this iteration.

4 Performance Metrics

Two metrics used in our experiment are Matching Score and Homography Estimation, which are identical to that used in [9]. In this section we introduce their definitions.

4.1 Matching Score

Matching Score measures the overall performance of interest point detector and descriptor. It measures the ratio of ground truth correspondences that can be recovered by detector and descriptor over the number of interest point proposed by the pipeline in the shared viewpoint region. Suppose the detector extracts $\|IP_{1}\|$ points from image $I_{1}$ and $\|IP_{2}\|$ points from image $I_{2}$ in the shared viewpoint region of $I_{1}$ and $I_{2}$. With descriptions of extracted points and a specific matching strategy, suppose $corr_{12}$ points in $I_{1}$ are correctly matched to points in $I_{2}$, and $corr_{21}$ points in $I_{2}$ are correctly matched to points in $I_{1}$. Then Matching Score for this pair of image is

[TABLE]

Same to [9], we use two-way nearest neighbor matching as the matching strategy. And Matching Score for entire dataset is obtained by averaging $M\text{-}score$ over all image pairs.

4.2 Homography Estimation

Homography Estimation measures the ability of an algorithm to estimate the homography that relates a pair of images by comparing it to ground truth homography. Denote the estimated homography as $\hat{H}$ and ground truth homography as $H$. Be identical to [9], Homography Estimation is defined with the error of four corners of one image onto the other. Denote the four corners of the first image as $c_{1}$, $c_{2}$, $c_{3}$ and $c_{4}$. We then apply the ground truth $H$ to get the ground truth corners in the second image which is denote as $c_{1}^{\prime}$, $c_{2}^{\prime}$, $c_{3}^{\prime}$ and $c_{4}^{\prime}$, and the estimated homography $\hat{H}$ to get the estimated corners in the second image which is denoted as $\hat{c_{1}}^{\prime}$, $\hat{c_{2}}^{\prime}$, $\hat{c_{3}}^{\prime}$ and $\hat{c_{4}}^{\prime}$. And the error of homography is defined as

[TABLE]

And the measure of Homography Estimation is defined as

[TABLE]

Here $\mathbb{I}$ is the indicator function, and $\epsilon$ is the threshold to judge the correctness of homography. Same to [9] we set $\epsilon=3$ in our experiment.

5 More Experimential Results

In main text we have demostrated results on image size $640\times 480$. In this subsection we give results of different methods on image size $320\times 240$. All experiment configures are same as that for image size $640\times 480$ except the maximum number of extracted interest points. We keep no more than 300 interest points for $320\times 240$ images. Table LABEL:tab:tabscore and LABEL:tab:tabhomo demostrate Matching Score and Homography Estimation of different results. And Figure 6 shows some matching results with image size $320\times 240$. Here our PN-i-64, PN-v-64 and PN-128 are identical to what in main text.