Anisotropic-Scale Junction Detection and Matching for Indoor Images

TL;DR

This paper introduces anisotropic-scale junctions (ASJs) that incorporate local geometric scales for improved detection and matching in indoor images, especially under viewpoint changes, outperforming existing methods.

Contribution

It presents a novel junction detection method that estimates independent scales for each branch using an ontraire model, enhancing indoor image matching accuracy.

Findings

Achieves state-of-the-art indoor image matching performance.

Effectively handles viewpoint variations and insufficient key-point features.

Provides more detailed geometric information through anisotropic scales.

Abstract

Junctions play an important role in the characterization of local geometric structures in images, the detection of which is a longstanding and challenging task. Existing junction detectors usually focus on identifying the junction locations and the orientations of the junction branches while ignoring their scales; however, these scales also contain rich geometric information. This paper presents a novel approach to junction detection and characterization that exploits the locally anisotropic geometries of a junction and estimates the scales of these geometries using an \emph{a contrario} model. The output junctions have anisotropic scales --- i.e., each branch of a junction is associated with an independent scale parameter --- and are thus termed anisotropic-scale junctions (ASJs). We then apply the newly detected ASJs for the matching of indoor images, in which there may be dramatic changes in viewpoint and the detected local visual features, e.g., key-points, are usually insufficiently distinctive. We propose to use the anisotropic geometries of our junctions to improve the matching precision for indoor images. Matching results obtained on sets of indoor images demonstrate that our approach achieves state-of-the-art performance in indoor image matching.

Tables

Table 1. Table 1: Average number of false detections in 1000 1000 1000 images generated by Gaussian white noises

$ϵ$	0.01	0.1	1	10	100	200
Avg. False	0.002	0.006	0.198	5.923	66.472	132.676

Table 2. Table 2: Comparison of different matching methods. The number of correct matches, number of total matches and the matching accuracy for the comparision with key-points matching results are reported in the first row. The results for key point matching approaches SIFT  [ 10 ] , Affine-SIFT  [ 43 , 13 ] , Hessian-Affine  [ 44 ] , EBR  [ 12 ] and IBR  [ 12 ] are list in the 3-th row to 7-th row. The average matching accuracy for all collected images is reported in the last column.

		(a)	(b)	(c)	(d)	(e)	(f)	(g)	(h)	(i)	(j)	(k)	(l)	Average accuracy
Ours (Junctions)	#correct	12	26	12	16	50	14	197	65	119	37	17	33	85.17%
	#total	12	29	13	20	60	15	214	69	121	45	19	40
	accuracy (%)	100.00	89.66	92.31	80.00	83.33	93.33	92.06	94.20	98.35	82.22	89.47	82.50
SIFT [10]	#correct	128	476	559	115	435	74	708	199	147	200	103	65	62.69%
	#total	206	700	839	222	652	287	770	261	330	299	191	161
	accuracy (%)	62.14	68.00	66.63	51.80	66.72	25.78	91.95	76.25	44.55	66.89	53.93	40.37
Affine-SIFT [13]	#correct	135	183	433	364	430	119	4141	1271	136	172	196	163	82.85%
	#total	141	240	519	480	549	133	4205	1326	240	263	224	264
	accuracy (%)	95.74	76.25	83.43	75.83	78.32	89.47	98.48	95.85	56.67	65.40	87.50	61.74
Hessian-Affine [44]	#correct	24	13	96	82	66	17	640	226	32	114	38	29	79.68%
	#total	26	34	132	105	109	18	671	248	63	144	41	49
	accuracy (%)	92.31	38.24	72.73	78.10	60.55	94.44	95.38	91.13	50.79	79.17	92.68	59.18
EBR [12]	#correct	0	0	10	0	0	0	64	20	28	14	0	0	32.56%
	#total	1	1	16	15	10	2	75	21	46	34	2	4
	accuracy (%)	0.00	0.00	62.50	0.00	0.00	0.00	85.33	95.24	60.87	41.18	0.00	0.00
IBR [12]	#correct	0	0	28	11	14	0	46	0	0	10	0	0	31.84%
	#total	4	9	39	16	25	9	63	12	10	21	8	5
	accuracy (%)	0.00	0.00	71.79	68.75	56.00	0.00	73.02	0.00	0.00	47.62	0.00	0.00

Table 3. Table 3: Comparison of different matching methods for line segment matching. The number of correct matches are counted by the rule that endpoints of corresponding line-segments are correct. We compare ASJ with state-of-the-art approaches LPI  [ 14 ] and LJL  [ 15 ] and report the number of correct matches, number of total matches and the matching accuracy in this table. The average matching accuracy is also compare in the last column.

		(a)	(b)	(c)	(d)	(e)	(f)	(g)	(h)	(i)	(j)	(k)	(l)	Average accuracy
Ours (Line segments)	#correct	15	30	14	26	85	21	349	121	232	49	27	48	71.55%
	#total	24	58	26	40	120	30	428	138	242	90	38	80
	accuracy (%)	62.50	51.72	53.85	65.00	70.83	70.00	81.54	87.68	95.87	54.44	71.05	60.00
LPI [14]	#correct	5	0	15	19	53	3	123	60	33	17	11	16	48.83%
	#total	9	0	18	29	90	9	193	102	59	38	15	40
	accuracy (%)	55.56	0.00	83.33	65.52	58.89	33.33	63.73	58.82	55.93	44.74	73.33	40.00
LJL [15]	#correct	8	24	26	37	148	4	221	113	129	50	26	22	52.95%
	#total	30	79	32	64	251	17	376	186	138	131	50	102
	accuracy (%)	26.67	30.38	81.25	57.81	58.96	23.53	58.78	60.75	93.48	38.17	52.00	21.57

Equations

$$ȷ=\left\{\mathbf{p},{\left\{r_i,{\theta }_i\right\}}_{i=1}^M\right\}$$

$$\begin{array}{r}{S}_{\mathbf{p}}(r,\theta ):=\{\mathbf{q}\in \Omega ;\mathbf{q}\ne \mathbf{p},\Vert \overrightarrow{\mathbf{pq}}\Vert \le r,\\ d_{2\pi }(\alpha (\overrightarrow{\mathbf{pq}}),\theta )\le \Delta (r)\}\end{array}.$$

$${\omega }_{\mathbf{p}}(r,\theta )=\sum _{\mathbf{q}\in S_{\mathbf{p}}(r,\theta )}{\gamma }_{\mathbf{p}}(\mathbf{q}),$$

$$\begin{array}{r}{\gamma }_{\mathbf{p}}(\mathbf{q})=\Vert \nabla \tilde{I}(\mathbf{q})\Vert \cdot \max (\left|\cos (\varphi (\mathbf{q})-\alpha (\overrightarrow{\mathbf{pq}}))\right|\\ -\left|\sin (\varphi (\mathbf{q})-\alpha (\overrightarrow{\mathbf{pq}}))\right|,0)\end{array},$$

$$t(ȷ):=\underset{m=1,\dots ,M}{\min }{\omega }_{\mathbf{p}}(r,{\theta }_m),$$

$$F(t;J(r,\theta ))=\mathbb{P}\{{\omega }_{\mathbf{p}}\ge t\}={\int }_t^{+\infty }d\left({\star }_{j=1}^{J(r,\theta )}p\right)$$

$$p(z)=\frac{1}{2}({\delta }_0(z)+\frac{2}{\sqrt{\pi }}{e}^{-\frac{z^2}{4}}\mathrm{erfc}(\frac{z}{2}))1_{z\ge 0}.$$

$$\mathrm{NFA}(ȷ):=\#\mathcal{J}(1)\cdot F(t;J(r,\theta ))\le \epsilon ,$$

$${\omega }_{\mathbf{p}}(r,\theta )={\int }_0^{r}dr{\int }_{\theta -\delta (r)}^{\theta +\delta (r)}{\gamma }_{\mathbf{p}}\left(\mathbf{p}+r\left(\cos \psi ,\sin \psi \right)\right)d\psi ,$$

$$\begin{array}{r}\frac{\partial {\omega }_{\mathbf{p}}(r,\theta )}{\partial r}=\sum _i{\gamma }_{\mathbf{p}}\left(\mathbf{p}+r\left[\begin{array}{c}\cos {\psi }_i\\ \sin {\psi }_i\end{array}\right]\right)\\ +{\delta }^{\prime }(r)\sum _i{\gamma }_{\mathbf{p}}\left(\mathbf{p}+r_i\left[\begin{array}{c}\cos (\theta -\delta (r))\\ \sin (\theta -\delta (r))\end{array}\right]\right)\\ +{\delta }^{\prime }(r)\sum _i{\gamma }_{\mathbf{p}}\left(\mathbf{p}+r_i\left[\begin{array}{c}\cos (\theta +\delta (r))\\ \sin (\theta +\delta (r))\end{array}\right]\right)\end{array},$$

$$S_r=\sum _{i=1}^m{X}_i+{\delta }^{\prime }(r)\sum _{i=1}^k(Y_i+Z_i),$$

$$f(t)=\frac{1}{\sqrt{4k{\sigma }^2\pi }}\exp \left(-\frac{(t-2k\mu {)}^2}{4k{\sigma }^2}\right)$$

$$\tilde{f}(t)=\frac{R^2}{\tau }f(-\frac{R^2}{\tau }t)=\frac{1}{\sqrt{2\pi \cdot 2k{\sigma }^2\frac{{\tau }^2}{R^4}}}\exp \left(-\frac{t+\frac{2\tau }{R^2}k\mu }{2\cdot 2k{\sigma }^2\frac{{\tau }^2}{R^4}}\right),$$

$${\mathbb{P}}_{\mathbf{p}}(r,\theta ):=\mathbb{P}\left(S_r\ge \frac{\partial {\omega }_{\mathbf{p}}}{\partial r}\right)={\int }_{\partial {\omega }_{\mathbf{p}}/\partial r}^{\infty }d\left({\star }_{i=1}^{M}p\right),$$

$$\mathrm{NFA}(r,\mathbf{p},\theta )=N_s\cdot {\mathbb{P}}_{\mathbf{p}}(r,\theta )\le \epsilon ,$$

$$\mathrm{NFA}(r,\mathbf{p},\theta )=\sqrt{NM}\cdot {\mathbb{P}}_{\mathbf{p}}(r,\theta )\le \epsilon ,\forall r\in (0,R].$$

$${ȷ}_1=\{{\mathbf{p}}_1,\{r_1^1,{\theta }_1^1\},\{r_1^2,{\theta }_1^2\}\},$$

$${ȷ}_2=\{{\mathbf{p}}_2,\{r_2^1,{\theta }_2^1\},\{r_2^2,{\theta }_2^2\}\}$$

$$r^{\ast }=\underset{i}{\max }{r}_i.$$

$${\omega }_{\mathbf{p}}(\theta )=\frac{1}{\sigma }\left(\sum _{\mathbf{q}\in S_{\mathbf{p}}(\theta )}\frac{{\gamma }_{\mathbf{p}}(\mathbf{q})}{\sqrt{n}}-\sqrt{n}\mu \right),$$

$$\begin{array}{r}{\tilde{\gamma }}_{\mathbf{p}}(\mathbf{q})={\omega }_{\mathbf{q}}^i\cdot \max (\left|\cos ({\theta }_{\mathbf{q}}^i-\alpha (\overrightarrow{\mathbf{pq}}))\right|-\\ \left|\sin ({\theta }_{\mathbf{q}}^i-\alpha (\overrightarrow{\mathbf{pq}}))\right|,0)\end{array}$$

$${\tilde{\omega }}_{\mathbf{p}}(r,\theta )=\sum _{\mathbf{q}\in S_{\mathbf{p}}(r,\theta )}{\tilde{\gamma }}_{\mathbf{p}}(\mathbf{q}),$$

$$\tilde{p}(z)=\frac{1}{2}{\delta }_0(z)+1_{z\ne 0}\frac{\sqrt{2}}{\sqrt{{\pi }^3}}{\int }_0^1\frac{1}{y}{e}^{-\frac{z^2}{2y^2}}\frac{1}{\sqrt{2-y^2}}dy,$$

$${\mathbb{P}}_{\mathbf{p}}(r,\theta ):=\mathbb{P}\left(S_r\ge \frac{\partial {\tilde{\omega }}_{\mathbf{p}}}{\partial r}\right)={\int }_{\partial {\tilde{\omega }}_{\mathbf{p}}/\partial r}^{\infty }d\left({\star }_{i=1}^M\tilde{p}\right).$$

$$\mathrm{NFA}(r_i^j,{\mathbf{p}}_i,{\theta }_i^j)=\sqrt{NM}\cdot {\mathbb{P}}_{\mathbf{p}}(r_i^j,{\theta }_i^j)\le \epsilon ,$$

$$\hat{\theta }=\arg \underset{\theta }{\max }\sum _{\mathbf{q}\in S_{\mathbf{p}}(r_{\theta },\theta )}{\gamma }_{\mathbf{p}}(\mathbf{q}),$$

$$\{\begin{array}{r}(r^1,{\theta }^1)<(r^2,{\theta }^2),if⟨{\theta }^1,{\theta }^2⟩<\pi \\ (r^1,{\theta }^1)<(r^2,{\theta }^2),if⟨{\theta }^1,{\theta }^2⟩>\pi \end{array}.$$

$$\begin{array}{r}{\mathbf{q}}_i=H{\mathbf{p}}_i,i=0,1,2\end{array},$$

$$H=\left[\begin{array}{ccc}{h}_1& h_2& h_3\\ h_4& h_5& h_6\\ 0& 0& 1\end{array}\right]$$

$$\mathcal{D}({ȷ}_n^P,{ȷ}_m^Q)=\tilde{D}(\mathcal{P}({ȷ}_n^P),\mathcal{P}({ȷ}_n^Q))+\tilde{D}(\mathcal{P}({ȷ}_m^Q),\mathcal{P}({ȷ}_m^P))$$

Full text

1 Introduction

Image correspondence is a key problem for many computer vision tasks, such as structure-from-motion [1, 2, 3, 4], object recognition [5, 6] and many others [7, 8]. The past decades have witnessed the big successes on that problem achieved by detecting and matching local visual features [9, 10, 11, 12, 13]. Although most of existing image matching algorithms relying on such local visual features perform well for images containing rich photometric information, e.g. outdoor images, they usually lose their efficiency on images that are less photometric and dominated by geometrical structures such as indoor images displayed in Fig. 1. In the indoor scenario, images are often dominated by low-texture parts and are with severe viewpoint changes, in which case it is reported to be more effective to make the correspondence of geometrical structures [14, 15] such as line segments [16, 17] and junctions [18].

The line segment matching problem has been studied in recent years since it can represent more structural information than key-points. Many algorithms match line segments by using either photometric descriptors with individual line segments [19] or the initial geometric relation [14, 15] to assist line segment matching. The approaches using pre-estimated epipolar geometry usually perform better than those of using photometric descriptors [19], but the epipolar geometry estimation still needs key-point correspondences in many situations. In the indoor scenes, due to the fact that descriptors for low-textured regions are not distinctive enough, it is very likely to produce unstable epipolar geometry for inferring the line segment matching. It is thus of great interest to develop elegant ways to make the correspondences of geometrical structures of images while get rid of the errors raised by the key-point correspondences, for finally achieving better matching of indoor images.

Alternatively, as a kind of basic structural visual features, junctions, have been studied as the primary importance for perception and scene understanding in recent years [20, 21, 22]. Being a combination of points and ray segments, junctions contain richer information than line segments, i.e. including a location and at least two ray segments (known as branches). Ideally, the information contained by a pair of junctions enables us to recover the correspondences between images up to affine transformations. However, due to the difficulties in the estimation of the endpoints of junction branches, most of junction detection algorithms [23, 24, 25, 26, 27, 28] concentrate on identifying the locations and orientation of branches while ignoring their length. This actually simplifies junctions as key-points and does not fully exploit their capabilities for image correspondences. To characterize the structure of junctions better, the detector ACJ [28] estimates scale invariant junction and it can be represented isotropically as a circle region with two or more dominant orientations. Every orientation represents a branch of junction and the radius of circle is equal to the shortest length among these branches. Although the orientation of branches is invariant with respect to viewpoint, it is not enough for estimating the affine transformation. Fortunately, if we can estimate the length of every branch, the affine transformation will be determined by a pair of junction correspondence.

Motivated by this, we are going to study for exploiting the invariance of junctions through estimating scale (length) of branches. For indoor images, the inherent scale for junctions usually are the length of some (straightforward) boundary for salient objects in images, which contains rich structure information and beyond local features. More precisely, we proposed an a-contrario approach that models the endpoints of a ray segment starting at given location with initial orientations, which check the proposed point if it should be a part of the ray segment according to number-of-false-alarms (NFA). When the points that belong to the ray segment occurs continuously until the continuity broken, the inherent scale for the ray segment is determined. In reality, the initial orientations are noised, we also optimize them with the junction-ness based on the a-contrario theory. Once the anisotropic scale is estimated for each branch (ray segment), the local homography can be estimated from any pair of junctions extracting from two different images. Theoretically, the correct correspondence produce reasonable local affine homography while incorrect correspondences generate local homographies in their own way. Considering the certainty of junction locations, the regions around location can be mapped by correct or incorrect affine homography. Correct homographies will map one image to another with minimal patch distortion. Comparing the regions with induced affine homography for a pair of junctions can check if the pair are correspondence. When the corresponding junctions are identified in image pairs, the results will produce more structure information. Our contributions in this paper are

• •

We extent the junction detector in [28] to anisotropic-scale geometrical structures, which can better depict the geometric aspect of indoor images.

• •

We developed an efficient scheme for making the correspondence of anisotropic-scale junctions. More precisely, as a detected anisotropic-scale junction provides at least three points, each pair of junctions in images can induce an affine homography. We finally present a strategy by induced homographies to generate accurate and reliable correspondences for the location and anisotropic branches of junctions simultaneously.

• •

We evaluate our method on challenging indoor image pairs, e.g. some of images are from the indoor image datasets used in [29, 30] and our results demonstrate that it can achieve state-of-the-art performance on matching indoor images.

The rest of this paper is organized as follows. First, the existing research related to our work is given in Sec. 2. In Sec. 3, the problem of detecting and matching junctions for indoor scene is discussed. Next to this section, an a-contrario approach for detecting anisotropic-scale junction is described. As for the junction matching, we design a dissimilarity in Sec. 5 to find the correspondences. The experimental results and analysis for our approach are given in Sec .6. Finally, we conclude our paper in Sec. 7.

2 Related works

In this section, we briefly review the existing approaches for junction detection and matching as well as geometrical structure matching for indoor images.

2.1 Junction detection

Detecting junction structure in images has been studied for years[31, 32, 33, 24, 34, 26, 27]. In the early stage, junction was studied as corner points [31, 32]. For the sake of recognition, the scale of junctions or other key-points also have been studied [10, 33, 28]. These approaches estimate the scale around junction locations by using scale space theories [35, 10, 9, 33] to handle the viewpoint changes across different images. Since these approaches determine the scale of interested points in very local area, their precision and discriminability will be lost quickly. Besides, these methods mainly focus on the localizations and scales of corner points while ignoring the differences between different type of junctions.

To overcome these shortcomings, the ACJ detector [28] was proposed to detect and characterize junctions with non-linear scale space. In this work, an a-contrario approach is proposed for determining the location and branches of junctions with interpretable isotropic scales, which characterizes the ray segments as junction branches and locations explicitly. The scales for detected junctions correspond to the optimal size at which one can observe the junction in the image.

Similar to junction detection, there is an elegant detector named edge based region (EBR) detector [12] for detecting affine invariant regions by estimating relative speed for two points that move away from a corner in both directions along the curve edges. This work can be regarded as a kind of junction detector in curve dominated images. The straight edges which are common in indoor scenario cannot be tackled in this way.

Although above mentioned approaches can extract junctions, their geometric representation is not exploited sufficiently. The scales estimated by these methods are local and insufficient for characterizing indoor scenes.

2.2 Junction matching

Junction matching has been attended since early years and shown promising matching accuracy[18, 36].

In [18], a model for estimating endpoints of junction branches is proposed which is very close to our work that estimating anisotropic scales for each branch. Differently, their approach [18] requires a roughly estimated fundamental matrix while our proposed method estimating anisotropic scales for each branch directly without fundamental matrix. For known fundamental matrix, the local homography between a pair of junctions can be estimated to produce more accurate results and refining epipolar geometry meanwhile [36]. These results are very related to recent approach for the hierarchical line segment matching approach LJL [15]. In this work, detected line segments are used to generate junctions with virtual intersections in the first stage. After that, junctions are regarded as key-points for matching initially. Finally the epipolar geometry induced from initial matching is used to estimate line segment correspondences with local junctions. The matching accuracy in fact relies on the descriptors of virtual intersections. Although their matching results are promising, the problem of estimating epipolar geometry need to other ways.

2.3 Indoor image matching with geometric structure

Most of indoor scenes can be described by using simple geometric elements such as points and line segments. As a combination of points and lines, junction is also a sort of useful geometrical structure for indoor scene. There has been many approaches such as Canny edge detector [37] and line segment detector (LSD) [16, 17] to extract line segments. LSD, which can produce more complete line segments than canny edge without any parameter tuning procedure, has been applied in many tasks such as line-segments matching [15] and 3D reconstruction [38]. Compared with key-points, line-segments can produce more complete result that contain the primary sketch for the scene.

Most of algorithms for line-segments matching rely on key-point correspondences. More precisely, key-points for an input image pair are firstly detected by using SIFT [10] or other detectors while estimating the epipolar geometry between the image pair by using RANSAC [39] and its variants. Based on the fundamental matrix $F$ induced by key-points matching, many approaches such as line-point-invariant (LPI) [14] and line-junction-line (LJL) [15] can match line segments correctly. LPI has ability to handle the relation between line-segments and matched key-points with viewpoint changes. LJL[15] method matches image pairs in multiple stages. In the first, detected line-segments are intersected with appropriate threshold to produce junctions and matching these intersections in the same way with key-points matching. Then, local homography are estimated for these junctions with the estimated fundamental matrices from key-points matching results. Although these approaches produce good performance in many cases, the matching results are in favor of matching lines instead of line-segments. Their results show that lines are matched while the endpoints of line segments are not matched very well. Except for the reason that the estimated epipolar geometry is sometime erroneous, there is a important reason for failure of line-segments matching that line segment detectors can not guarantee that the line-segments are consistent across imaging condition varying. In many situations, a line segment $l_{A}$ detected in image $I_{A}$ might be decomposed to two or more collinear line segments $l_{B}^{1},\ldots,l_{B}^{k}$ in another image $I_{B}$. In this case, the results of line matching can be regarded as correct if the line in $l_{A}$ is corresponding with $l_{B}^{1},\ldots,l_{B}^{k}$. However, in the aspect of line-segments matching, there exists no correct corresponding line-segments for $l_{A}$ in image $I_{B}$. On the other hand, the existing line segment matchers rely on the results of key-point matching. Once the key-points matching failed or inaccurate, the induced result of line segment matching will be affected in some extent.

3 Problem Statement

3.1 Junction Model

The early researches for junction detection usually focus on the orientations of branches and the locations while ignoring the length or scale of each branch. Even though the junction locations and orientation of branches are important to depict geometric structure for images, lacking scale of branches limits their performance for image matching. Motivated by this, we want to propose a new junction model for characterizing junction better. We define the our junction model by considering the endpoint of each branch. Since the length of every branch is possible to be different, we call our model as anisotropic-scale junction. As a special case, junction model with isometric branches is called isotropic-scale junction.

Definition 1** (Anisotropic-scale junction)**

An anisotrpic-scale junction with $M$ branches starting at the same location $\bm{p}$ is denote as

[TABLE]

where $r_{i}$ and $\theta_{i}$ are the scale and orientation for $i$-th branch, $M$ is the number of branches.

Fig. 2 provides an example for the difference between anisotropic-scale (left) and isotropic-scale (right) junctions is shown. The isotropic-scale junction is actually a special case for the anisotropic model when the length of all branches are identical.

3.2 Detecting Junction Locations and Isotropic Branches

Since junction is formed by several intersected line segments, the problem of localizing the intersection and identifying the normal angle of these line segments is easier to be focused. Once the isotropic junction model is defined, this problem becomes a template matching problem. Based on this idea, Xia et al. exploited the junction-ness for branches with given scale $r$ and orientations $\theta_{i}$ and then an a-contrario approach is derived to determine meaningful junctions for input images [28]. The junction-ness for given scale $r$ and orientation $\theta_{i}$ actually contains the neighbor information of normalized gradient. Different from points, the neighborhood for a given scale $r$ and $\theta_{i}$ is a sector. As shown in the left of Fig. 2, the dark area with $\theta_{1}$ represent the sector neighbor of the branch. The sector neighbor for given location $\bm{p}$, scale $r$ and orientation $\theta$ can be denoted mathematically as

[TABLE]

where the $\Delta(r)$ is defined as $\frac{\tau}{r}$ with some predefined parameter $\tau$, $\Omega$ is the domain of input image, $d_{2\pi}$ is the distance along the unit circle, defined as $d_{2\pi}(\alpha,\beta)=\min\left(|\alpha-\beta|,2\pi-|\alpha-\beta|\right)$ and $\alpha(\vec{\bm{pq}})$ is the angle of the vector $\vec{\bm{pq}}$ in $[0,2\pi]$.

Since a junction is formed by edges and corner points, the normal angle for gradient should be consistent with the orientation of branches. Followed with this idea, if most of points $\bm{q}\in S_{\bm{p}}(r,\theta)$ have close normal angles with orientation $\theta$, the corresponding scale $r$ and orientation $\theta$ should be meaningful to be a branch of the junction. For a given sector $S_{\bm{p}}(r,\theta)$, the junction-ness can be measured by

[TABLE]

and $\gamma_{\bm{p}}(\bm{q})$ is the pairwise junction-ness with

[TABLE]

where the $\left\|\nabla\tilde{I}(\bm{q})\right\|$ is the norm of normalized gradient at point $\bm{p}$, $\phi(\bm{q})$ for pixel $\bm{q}$ is defined as $\phi(\bm{q})=(\arctan\frac{I_{y}(\bm{q})}{I_{x}(\bm{q})}+\pi/2)~{}modulo~{}(2\pi)$, $I_{x},I_{y}$ are the partial derivative of input image in $x$ and $y$ direction.

For the isotropic scale junction with two or more branches, the minimal junction-ness for one of the branches is used to describe the junction-ness for the entire junction with the equation (5)

[TABLE]

where the number $M$ and $m$ represent the total number of branches and branch index for the junction $\jmath$.

3.3 Analysis for Estimating Anisotropic-scale Branches

Although the equation (5) measures junction-ness for a given junction, it does not contain any anisotropic scale for branches. Such definition of junction-ness only keeps the information that each branch’s scale $r_{i}$ is larger than $r$ and it cannot be used for handling more sophisticated transformations such as affine transform and projective transform. To overcome this problem, we define the anisotropic-scale junctions with independent scales in Def. 1. The difference between isotropic-scale junction and the anisotropic-scale one can be observed in Fig. 2. It is easy to see that the junction-ness for entire junction defined in Eq. (5) cannot be used to exploit independent scales $r_{i}$. Fortunately, the isotropic-scale junctions detected by ACJ [28] is meaningful and the problem of estimating scale $r_{i}$ and orientation $\theta_{i}$ can be simplified to estimating only scale $r_{i}$ with given location $\bm{p}$ and orientation $\theta_{i}$. In other words, for the detected isotropic junctions, we need to exploit a robust method to estimate the length of corresponding ray segment with specific orientation $\theta$.

One plausible way to model the unknown scale with respect to given location $\bm{p}$ and orientation $r$ is that simply modify the junction-ness defined in Eq (3) to $\omega_{\bm{p}}(r)$ with specific $\theta$. Then, the a-contrario approach in [28] seems to be feasible to check whether the scale $r$ is $\varepsilon$-meaningful. The corresponding cumulative distribution function (CDF) used to get $\varepsilon$-meaningful scale $r$ can be formulate to

[TABLE]

where the $p$ represents the distribution of random variable $\omega_{\bm{p}}(r,\theta)$ with

[TABLE]

$J(r,\theta)$ is the number of pixels in corresponding sector neighbor and the operator $\mathop{\star}_{j=1}^{J(r,\theta)}$ produces the convolutional probability density function (PDF) with $J(r,\theta)$ times, which actually represents the random variable of $\omega_{\bm{p}}(r,\theta)$. The $\varepsilon$-meaningful scale for given orientation and location can be determined by the inequality

[TABLE]

where $\#\mathcal{J}(1)$ is the number of test for junctions with $1$ branch.

However, the NFA defined in Eq. (8) has to face the fact that there exist several junctions in indoor images which have extremely large scale branches. This fact would lead to the above inequality disabled. To illustrate this problem, we studied the relationship between convolution times $J(r,\theta)$ with the minimal junction-ness that can make the probability $F(t;J(r,\theta))=0$. As shown in Fig. 3, if the value of $\omega_{\bm{p}}(r,\theta)$ is greater than $\frac{J(r,\theta)}{2}$, the probability of $F(t;J(r,\theta))$ will be equal to [math] constantly, which may cause the inequality degenerated to $0\leq\varepsilon$. In fact, the pairwise junction-ness defined in Eq (4) can reach to $1$ and then the $\omega_{\bm{p}}(r,\theta)$ will be equal to $J(r,\theta)$. Therefore, the junction-ness in [28] is infeasible to model the unknown scale.

4 An a-contrario model for anisotropic-scale junction detection

To solve the problems addressed in Sec. 3, we derive a differential junction-ness model for depicting scale with given location and orientation. Since the scale $r_{i}$ for each branch of junction $\jmath$ is irrelevant, we just model the endpoint of each branch independently.

4.1 Differential Junction-ness Model

Suppose the isotropic junctions have been detected in a small scale $r_{0}$, the inherent scales of branches will be greater than $r_{0}$. If we increase the scale $r_{0}$ to larger $r_{1}$, though the junction-ness is still larger, the error $\omega_{err}=\omega_{\bm{p}}(r_{1},\theta)-\omega_{\bm{p}}(r_{0},\theta)$ will not be increased significant. A reasonable way to recognize the un-significant variation is to study the variation of $\omega_{\bm{p}}(r,\theta)$ with respect to $r$ increased. Here, we first reformulate the junction-ness for a branch (3) in continuous form. The junction-ness for position $\bm{p}$, scale $R$ and orientation $\theta$ is

[TABLE]

where the $\delta(r)$ is the angle width for given scale, here, we select $\delta(r)=\frac{\tau}{r}$. The descrete partial derivative $\frac{\partial\omega_{\bm{p}}(r,\theta)}{\partial r}$ is given by

[TABLE]

where $\psi_{i}$ is the $i$-th sample angle in the range $\left[\theta-\delta(r),\theta+\delta(r)\right]$ and $r_{i}$ is the $i$-th sample point in the range $[0,r]$.

4.2 Null Hypothesis and Distribution

After the differential junction-ness model built, we need to find a robust way to check if the value of ${\partial\omega_{\bm{p}(r,\theta)}}/{\partial r}$ for specific $r$ is significant enough. One way to achieve this goal is developing an a-contrario approach to control the threshold automatically. Since our work is an extension of ACJ [28], the null hypothesis here should be same, we say the variables $\left\|\nabla\tilde{I}(\bm{q})\right\|$ and $\phi(\bm{q})$ follow the null hypothesis $\mathcal{H}_{0}$ if

$\forall\bm{q}\in\Omega$, $\left\|\nabla\tilde{I}(\bm{q})\right\|$ follows a Rayleigh distribution with parameter 1; 2. 2.

$\forall\bm{q}\in\Omega$, $\phi(\bm{q})$ follows a uniform distribution over $[0,2\pi]$; 3. 3.

All of the random variables $\left\|\nabla\tilde{I}(\bm{q})\right\|,\phi(\bm{q})\}_{\bm{q}\in\Omega}$ are independent each other.

According to the dicussion in [28], every $\gamma_{\bm{p}}(\bm{q})$ follows the distribution (7) independently. The random variable $\frac{\partial\omega_{\bm{p}(r,\theta)}}{\partial r}$ follows the distribution of the random variable

[TABLE]

where the random variable $X_{i},Y_{i},Z_{i}$ follow the distribution in equation (7) , $m$ is the number of sampling points for $\psi_{i}$ and $k$ is the number of sampling points for $r_{i}$. The function $\delta^{\prime}(r)$ will be very small for reasonable $r$ (for example, $r\geq 4$ induced $|\delta^{\prime}(r)|=\frac{\tau}{r^{2}}\leq\frac{\tau}{16}$) since the parameter $\tau$ should have small values. Hence, the random variable could be approximated with $S_{r}\approx\sum_{i=1}^{m}X_{i}$ for computational simplicity. In practice, $k$ is larger than 10 and therefore the PDF of $\sum_{i=1}^{k}(Y_{i}+Z_{i})$ can be apprixmated accurately by using the Central Limit Theorem as

[TABLE]

where $\mu$ and $\sigma^{2}$ are the expectation and variance of (7). The PDF of $\delta^{\prime}(R)\left(\sum_{i=1}^{k}Y_{i}+Z_{i}\right)$ is

[TABLE]

which is the Gaussian distribution with mean $-\frac{2\tau}{R^{2}}k\mu$ and variance $2k\sigma^{2}\tau^{2}/R^{4}$. Meanwhile, the random variable $\sum_{i=1}^{m}X_{i}$ follows $\mathcal{N}(m\mu,m\sigma^{2})$. Therefore, the random variable $S$ follows the distribution $\mathcal{N}(m\mu-2k\mu\frac{\tau}{R^{2}},m\sigma^{2}+2k\sigma^{2}\tau^{2}/R^{4})$ approximately.

The probability $\mathbb{P}\left(S_{r}\geq\frac{\partial\omega_{\bm{p}}(r,\theta)}{\partial r}\right)$ for given $r$ and $\theta$ follows the distribution $f(t)$

[TABLE]

describes the fact that scale cannot be increased with a sufficient small incremental at $R$ along orientation $\theta$ under the hypothesis $\mathcal{H}_{0}$. The smaller probability $\mathbb{P}_{\bm{p}}(r,\theta)$ is, the more confident that scale $r$ is a reasonable scale. The small probability $\mathbb{P}(r,\theta)$ means that the point $\bm{p}+r\cdot[\cos\theta,\sin\theta]$ belongs to the branch with high possibility. Ideally, the existed branch should produce a series small probability in a interval $[r_{0},r_{1}]$. Then, the (maximum) scale of the branch should be defined as $r_{1}$. We use the probability $\mathbb{P}_{\bm{p}}(r,\theta)$ to check if the point $\bm{p}_{r}^{\theta}=\bm{p}+r\cdot[\cos\theta,\sin\theta]^{T}$ belongs to the branch.

4.3 Number of Test and Number of False Alarms

In last subsection, we conclude that sufficient small probability of $\mathbb{P}_{\bm{p}}(r,\theta)$ indicates that the point with certain direction $\theta$ and radius $R$ belongs to the branch more probably. The definition of sufficient probability need to be cleared. According to the Helmholtz principle, we bound the sufficient probability with the expectation of the number of occurrences of this event is less than $\varepsilon$ under the a-contrario random assumption [40] with

[TABLE]

where the $N_{s}$ denotes the number of occurrences of the point occurs along the given location and orientation. Since the location and orientation of the branch are known, expected number of false alarms should be smaller than $\sqrt{NM}$ where $N$ and $M$ are the number of rows and columns of the corresponding image. When the point $\bm{p}_{r}^{\theta}$ $\forall r\in(0,r_{1}]$ rejects the hypothesis $H_{0}$, the scale of the branch should be $r_{1}$. The scale $R$ is called as the maximum (meaningful) scale of the branch if the scale $R$ is the maximum scale that satisfies inequality

[TABLE]

Usually, the $\varepsilon$ is defined as $1$, which means the expected Number of False Alarm is not larger than 1.

4.4 Scale Ambiguity for Branches

Junctions are located at the intersections of line segments. Suppose there exist two junctions

[TABLE]

[TABLE]

where the two-tuples $\{\{r_{i}^{j},\theta_{i}^{j}\}\}$ denotes the scale and orientation for the $j$-th branch of the $i$-th junction and $\bm{p}_{i}$ is location of the $i$-th junction. In the case that the junction $\jmath_{2}$ is located at $\bm{p}_{2}=\bm{p}_{1}+r_{1}^{1}\left[\cos\theta_{1}^{1},\sin\theta_{1}^{1}\right]^{T}$ and $\theta_{1}^{1}=\theta_{2}^{1}$, the scale ambiguity occurs since the line segment $\bm{p}_{1}\bm{p}_{2}$ and the branch $\{r_{2}^{1},\theta_{2}^{1}\}$ are co-linear. The scale of the first branch of $\jmath_{1}$ can be regarded as either $r_{1}^{1}$ or $r_{1}^{1}+r_{2}^{1}$. For example, there are two junctions $\jmath_{1}$ and $\jmath_{2}$ located at $\bm{p}_{1}$ and $\bm{p}_{2}$ respectively in the Fig. 4. The branch along the direction of ${\bm{p}_{1}\bm{p}_{2}}$ for $\jmath_{1}$ and $\jmath_{2}$ are co-linear with the line segment marked as red. For the branch of $\jmath_{1}$, its scales are $\left\|\bm{p}_{2}-\bm{p}_{1}\right\|$, $\left\|\bm{p}_{3}-\bm{p}_{1}\right\|$ and $\left\|\bm{p}_{4}-\bm{p}_{1}\right\|$ while the scales of the branch of $\jmath_{2}$ are $\left\|\bm{p}_{3}-\bm{p}_{2}\right\|$ or $\left\|\bm{p}_{4}-\bm{p}_{2}\right\|$. To eliminate the ambiguity, we define the scale for a branch as follow

Definition 2** (Scale of a branch)**

Suppose there exist a branch starting at point $\bm{p}$ in the direction $\theta$, the possible salient scales are $r_{1},r_{2},\ldots,r_{m}$, we define the scale of this branch $r^{*}$as

[TABLE]

The branch with such scale is more stable and more global than other features. However, there exist some challenges to estimate such scales from images. Most existing approaches and the model proposed in Sec. 4.3 estimate the line segment or branches based on orientations of level-lines extracted from the gradient of image [16]. The line segment detected from the image in Fig. 4 could be either $\bm{p}_{1}\bm{p}_{2}$ or $\bm{p}_{1}\bm{p}_{3}$ since the level-line around the points $\bm{p}_{2}$ have probability to aligned with the orientation of vector $\bm{o}_{2}-\bm{p}_{2}$, which will lead to the line segment that are co-linear with the branch of $\jmath_{1}$ across the point $\bm{p}_{2}$ to $\bm{p}_{3}$ or $\bm{p}_{4}$. When the viewpoint changed, illumination varied or noise increased, the orientations of level-lines around $\bm{p_{2}}$, $\bm{p}_{3}$ and $\bm{p}_{4}$ will be changed with uncertainty. Then, the scale cannot be estimated robust for different imaging conditions.

Fortunately, the inherent property for location of junctions is stable whatever the imaging condition is. Although the orientations of level-lines around the locations of junctions will change with uncertainty, most of them are still aligned to one of the lines that intersects the junction. Motivated by this, we use the very local isotropic-scale junctions in a small neighbor(e.g. $5\times 5$ or $7\times 7$ window size) instead of gradient field and level-lines. For a pixel in an image, we calculate the junction-ness for different orientations in a small neighbor according to ACJ [28] algorithm as

[TABLE]

where $S_{\bm{p}}(\theta)$ is defined in (2) with fixed radius (eg. $r=3,5,7$), $n$ is the cardinal number of set $S_{\bm{p}}(\theta)$. $\mu$ and $\sigma^{2}$ are the mean and variance defined in (13). Then, we leverage the non-maximal-suppression (NMS) [41] to obtain the very local junctions and filter out branches for these junctions with non-meaningful NFA values according to (8). These very local junctions are denoted as $\jmath_{\bm{p}}=\{\theta_{\bm{p}}^{i},\omega_{\bm{p}}^{i},{\rm NFA}_{\bm{p}}^{i}\}_{i=1}^{K}$, where the $\omega_{\bm{p}}^{i}$ and the ${\rm NFA}_{\bm{p}}^{i}$ is the strength and corresponding NFA value for branch with $\theta_{i}$ orientation. In the case that pixel $\bm{p}$ is on (around) an edge, there will be two $\theta_{i}$ that align to the orientation of this edge up to $\pm\pi$. If the pixel $\bm{p}$ is around another junction, there will be multiple orientations aligned with different branches of this junction. Meanwhile, we incorporate the strength $\omega_{\bm{p}}^{i}$ instead of the norm of (normalized) gradient with into the a-contrario model proposed in Sec. 4.3 with modified probabilistic distribution.

4.5 Modified Probabilistic Distribution

For the sake of estimating scale for a branch with definition 2, the functions $\gamma_{\bm{p}}$ and $\omega_{\bm{p}}(r,\theta)$ measuring the junction-ness should be changed to

[TABLE]

and

[TABLE]

where the index $i$ in Eq. (16) is the orientation $\theta_{\bm{q}}^{i}$ that is most close to $\theta$.

According to the Central Limit Theorem(CLT), the random variable $\omega_{\bm{p}}^{i}$ follows the Gaussian distribution with mean [math] and variance $1$, the distribution for $\tilde{\gamma}_{\bm{p}}(\bm{q})$ is

[TABLE]

Then, the null Hypothesis discussed in Sec. 4.2 is updated to

[TABLE]

4.6 Junction Detection

So far, the a-contrario approach for anisotropic scale estimation is derived. For an input image, isotropic junctions and local junctions for each pixel are firstly detected by ACJ [28] for initialization. The results for junctions are denoted as $\{\jmath_{i}\}_{i=1}^{N}$ and local junctions at fixed small scale (usually $r=3,5,7$) for every pixels are $\jmath_{\bm{p}}=\{\theta_{\bm{p}}^{i},\omega_{\bm{p}}^{i},{\rm NFA}_{\bm{p}}^{i}\}_{i=1}^{K}$ where $\bm{p}$ is the coordinate of a pixel.

We estimate the scale $r_{i}^{j}$ for branch $\theta_{i}^{j}$ according to the Number of False Alarm

[TABLE]

where the probability $\mathbb{P}_{\bm{p}}(r,\theta)$ is the updated version in Eq. (19). The scale $r_{i}^{j}$ is searched starting at $r_{i}$ until the NFA is larger than $\varepsilon$.

The accuracy for orientations of branches detected by ACJ [28] is depend on the scale $r_{i}$ which is bounded by a predefined parameter and hence noised. The scales for ASJ is more sensitive to the noise which should be refined. A branch with the most accurate orientation $\theta$ should have the maximum junction-ness with the scale $r_{\theta}$, we optimize the objective function

[TABLE]

to refine the orientation for $\theta$ and check the branch with orientation $\hat{\theta}$ and scale $r_{\hat{\theta}}$ is $\varepsilon$-meaningful branch.

5 ASJ Matching for Indoor Images

Since the ASJs contain rich geometric structure informations represented by the anisotropic scales, we are going to study the matching method taken full advantage of ASJs. For a pair of junction $\jmath^{P}$ and $\jmath^{Q}$ detected from images $I^{P}$ and $I^{Q}$, the homography can be estimated by the points set that contain their locations and endpoints for branches, which can be used to compare junctions for correct correspondences. Since there exist $L$-junctions, $Y$-junctions and $X$-junctions in an image and the type of a junction might be different across images because of occlusion, the homography estimated from a pair of junctions might be invalid. Fortunately, whatever the type of junction is, the location can be intersected from any two of branches that are not parallel each other, which is saying that a junction with more than two branches can be decomposed two several $L$-junctions. Without saying, the $L$-junction with two branches that their orientation $\theta^{1}$ and $\theta^{2}$ are equal up to $\pi$ should be filtered out. After decomposing and filtering, the detected in an image are all $L$-junctions.

The perspective effects are typically small on a local patch [42], which can be approximated by affine homography. We use a pair of $L$-junctions to estimate such homographies. Suppose there are $N^{P}$ and $N^{Q}$ decomposed $L$-junctions in image $I^{P}$ and $I^{Q}$, denoted as $\{\jmath_{i}^{P}\}_{i=1}^{N^{P}}$ and $\{\jmath_{i}^{Q}\}_{i=1}^{N^{Q}}$ respectively. If a pair of junctions $(\jmath_{n}^{P},\jmath_{m}^{Q})$ are matched, an affine homography would be induced once the orientations are determined. In order to derive a unique affine homography, we define the partial order for two branches $(r^{1},\theta^{1})$ and $(r^{2},\theta^{2})$ of a $L$-junction as

[TABLE]

Every junction need to be sorted by the order defined above. The affine homography for a pair of junction $\jmath_{n}^{P}$ and $\jmath_{m}^{Q}$ are estimated by using DLT (Direct Linear Transform) with their locations and endpoints for the branches. More precisely, we solve the equations

[TABLE]

where $\bm{p}_{i}$ and $\bm{q}_{i}$ are the homogeneous representation of locations and two branches for $\jmath_{n}^{P}$ and $\jmath_{m}^{Q}$ respectively. The matrix $H$ is

[TABLE]

represents the affine transform induced by $\bm{p}_{i}$ and $\bm{q}_{i}$ for $i=0,1,2$.

From the image pair $(I^{P},I^{Q})$, there can be $N^{P}\times N^{Q}$ affine homographies, denoted by $H_{n,m}$, which maps the $n$-th junction in $I^{P}$ to $m$-th junction in $I^{Q}$. For correct correspondence , the matrix $H_{n,m}$ will map the image $I^{P}$ to $I^{Q}$ accurate around the location of junctions while the mismatch will map the image $I^{P}$ only correct at the endpoints and locations but erroneous at other positions. For the sake of saving computational resource, we just map a patch $\mathcal{P}(\jmath_{n}^{P})$ around $\jmath_{n}^{P}$ to $\mathcal{P}(\jmath_{n}^{Q})$ in $I^{Q}$ and map $\mathcal{P}(\jmath_{m}^{Q})$ to $\mathcal{P}(\jmath_{m}^{P})$ in $I^{P}$ by using matrix $H_{n,m}$ and its inverse $H_{n,m}^{-1}$. Then, the distance between two features $\jmath_{n}^{P}$ and $\jmath_{m}^{Q}$ are measured by

[TABLE]

where the distance $\tilde{D}$ are the distance between two patches calculated by raw patches, SIFT descriptor or other descriptors.

Benefiting with the homographies induced by ASJ, the distance between original patches and mapped patches for correct correspondence is usually very small while larger for incorrect correspondence, we can use ratio test proposed in [10] to filter out the incorrect correspondence.

6 Experimental Analysis

This section illustrates the results and analysis for ASJ detection and matching routines with comparison to existing approaches for junction detection, junction matching, key-points matching and line segment corresponding. In our experiments, we first detect anisotropic-scaled junctions by relying on the procedures presented in Section 4, and then make the correspondence of junctions with the affine homography induced by these semi-local geometrical structures.

6.1 Stability and Control of the Number of False Detection

The a-contrario approaches detect meaningful events controlled by the threshold $\epsilon$: it bounds the average number of false detections in an image following null hypothesis. In this subsection, we check the average number of false detections in Gaussian noise image and illustrate the results of detected ASJs with fixed threshold $\epsilon$.

Experimentally, we generate $1000$ random images with $256\times 256$ pixels which follow standard Gaussian distribution independently pixel-wised. For each pixel, we generate an orientation randomly from the uniform distribution in the interval $[0,2\pi)$ and estimate scale at this pixel with the orientation. Ideally, there is no meaningful line-segment structure appeared in random images but might be detected mistakenly, which are counted in number of false detection averagely. If the number of false detection can be controlled by the NFA proposed a-contrario approach, the approach would be identified as correct a-contrario approach.

The average number of false detections in $1000$ Gaussian noise images are reported in the Tab. 1. The value of NFA are varied in our experiments from $10^{-2}$ to $10^{2}$ and the corresponding average number of false detections are upper bounded by the NFA.

6.2 Comparison with ACJ

It is necessary to compare the repeatability for our proposed ASJ with ACJ since we extend the a-contrario model for scale estimation to discuss their difference. Following with the baseline experiments proposed in [28], these images are firstly zoomed with different factors to form the image sequences with scale change. Then, the ASJ and ACJ are performed on these image sequences to detect the junctions. The repeatability for ACJ is discussed in [28], however, their definition for corresponding junction just concentrates on the location and branch of junctions while ignoring the scale coherence. Therefore, we are going to define the corresponding ACJ and ASJ with scale information here. For the original image $I_{0}$ and the scaled image $I_{s}=s(I_{0})$, the corresponding ACJ junctions should have close locations, branch orientations as well as scales. Meanwhile, two junctions with different number of branches cannot be identified as correspondence. More precisely, we define two ACJ junctions $\jmath_{1}=\{\bm{p}_{1},r_{1},\{\theta_{i}\}_{i=1}^{M}\}$ and $\jmath_{2}=\{\bm{p}_{2},r_{2},\{\theta_{i}^{s}\}_{i=1}^{M}\}$ detected in $I_{0}$ and $I_{s}$ if they follow

[TABLE]

[TABLE]

[TABLE]

where the angular distance $d_{2\pi}(\theta,\theta^{\prime})=\min(\left|\theta-\theta^{\prime}\right|,2\pi-\left|\theta-\theta^{\prime}\right|)$. Similar to the above, the correspondence for two junctions $\jmath_{1}=\{\bm{p}_{1},\{r_{i},\theta_{i}\}_{i=1}^{M}\}$ and $\jmath_{2}=\{\bm{p}_{2},\{r_{i}^{s},\theta_{i}\}_{i=1}^{M}\}$ detected by ASJ should satisfy the inequalities (24), (26) as well as

[TABLE]

In this experiment, the set of scale factors is $\left\{1.0,0.9,0.8,\ldots,0.3\right\}$ and the results are shown in Fig. 5. Observing the repeatability curve, our proposed ASJ performs better than ACJ. The repeatability rate reported in [28] is higher, however, it just demonstrate the accuracy of locations and orientation of branches. In our experiment, the scale difference are also considered here.

As reported in [28], the scale of ACJ represents the length of shortest branch and it is roughly linear through the scale factors[28]. Theoretically, if a detected ACJ has scale $r$ in original image, its correspondence in the scaled image $s(I)$ should be close enough to $s\cdot r$. However, the upper bound of scale is required for ACJ algorithm as input and it is recommend to be set as in the range of $[12,30]$ constantly[28] for the sake of computational speed. As a matter of fact, the junctions in indoor images usually have large scale branches and they cannot be bound with a relative small constant.

To demonstrate this fact, we compare the detected junctions in Fig. 6. In this experiment, the junctions are detected by ACJ firstly in original image $I$ and scaled image $s(I)$ with the factor $s$ firstly. Then we find the corresponding ACJ in the image pair by using the inequalities (26) and (24) while ignoring the inequality (25). For the sake of comparing the scale of junctions with respect to factor $s$, all the correspondences are shown with colored circle. In Fig. 6, a correspondence of $\jmath_{2}={\bm{p}_{2},r_{2},\{\theta_{i}^{s}\}_{i=1}^{M}}$ in image $s(I)$ which has scale $r_{1}$ is shown with a yellow circle with the radius $r_{1}\cdot s$. The red circle and green line segments present the junction $\jmath_{2}$. We can find out that there exist several correspondences which do not have consistent scales. If a junction is formed by several line segments of which lengths are more than $1/s$ time of maximal radius threshold of ACJ, the scale of junction will be equal to the threshold in the original image. When the image is zoomed with factor $s$, the scale will not be decreased since it is still larger than the threshold. This is the reason why the repeatability is lower when we use the inequality (25) to calculate it.

In the final of this subsection, some example results of ASJ detector for indoor images are shown in Fig. 7. The anisotropic-scale junction are shown in the middle column and the results of ACJ are listed in the right column. Observing the results, we can find that ASJ has the ability to detect more geometric structure than ACJ. The anisotropic-scale branches of a junction can depict the layout of indoor scenes. By contrast, the results of ACJ just represent the very local information. For example, there are several rectangles in the Fig. 7, our ASJ can produce the boundary of the rectangle while ACJ just detect the corner points and orientations around the corners of rectangle.

6.3 ASJ Matching

In order to evaluate our approach, we collect more than 100 images to perform our proposed approach ASJ. Some of the collected images are from indoor 3D reconstruction dataset [30, 29] while others are taken by ourselves. As shown in Fig. 8, the collected images are less texture than natural images. Some of them contain large viewpoint changes and indistinct texture repeated regions such as Fig. 8(b), Fig. 8(i) and Fig. 8(l).

We define that two junctions are matched, only if the junction centers and orientations of branches are corresponding. In this sense, our matching result is somewhat beyond of local features and can be compared with existing approaches in different settings:

It is comparable to keypoint matching methods, if we regard junctions as a specific corner points with two orientations;

• -

It is also comparable to line segment matching ones, if we take junctions as several intersecting line segments.

For key-points matching, we compare the results of matched junctions with SIFT [10], Affine-SIFT [43, 13], Hessian-Affine [44], EBR and IBR in [12].

Meanwhile, we compare maching accuracy with existing approaches LPI [14] and LJL [15] for matched line segments that measures the proportion of matches if their endpoints are corresponding. This rule is more strict for assessing line segment matching results. Interestingly, the approaches LPI [15] and LJL [15] use the epipolar geometry without outliers to assist their line segment matcher, our proposed method without epipolar geometry achieves better accuracy.

The implementation for Affine-SIFT [43, 13], Hessian Affine [44], LPI [14] and LJL [15] are getting from authors’ homepage. EBR and IBR [12] are got from VGG’s website111http://www.robots.ox.ac.uk/~vgg/research/affine/descriptors.html#binaries. The version of SIFT detector is provided by VLFeat222http://www.vlfeat.org. The descriptor used in our experiment is SIFT and the mismatches are filtered according to the ratio test with threshold $1.5$ for ASJ , SIFT [10], Hessian-Affine [44], EBR [12] and IBR [12] by comparing the $\ell_{2}$-distance, which is the default threshold for computing matches from descriptor in VLFeat. Remarkably, the implementation of Affine-SIFT [43, 13] provided by its authors use threshold $1.33$ since they calculate the distance with $\ell_{1}$ norm and we keep it unchanged. Since the released code for Affine-SIFT [43, 13] produce the matched result with outliers filtering, we remove this procedure in all fairness, which makes the results in our experiment different from the released executable program. All of parameters for compared approaches are the default value which is provided by their authors.

6.3.1 Matching results for key-points matching

As shown in Tab. 2, our proposed feature ASJ is compare with most widely used feature detectors. In the sense for key-points matching, we regard an ASJ as a key-point with two specific orientations. The matching accuracy for ASJ is better than other key-points matches in most cases. Representatively, in Fig. 8(i), the indistinct repeated region in chessboard are matched very well with the accuracy $98.35\%$ since ASJs makes corner points contain more global information than other approaches, which represents the relative position with meaningful orientations in images.

Comparing with the most related approach EBR and IBR [12], our proposed approach ASJ handles straight edges in a better way which can produce more key-points and more correct correspondences. In many cases as shown in Tab. 2, the results of EBR and IBR illustrate their limitation in indoor images which are dominated by straight edges.

In the aspect of absolute number of correct matches, ASJ is less than other approaches significantly. The approaches matching most number of correct matches are Affine-SIFT and SIFT. Since the junctions detected in indoor images represents the meaningful junctions in the scene, the fact that absolute number is less than SIFT key-points is not surprising. Nevertheless, ASJ represents the structure information compactly for scenes than key-points. To illustrate this, we plot the correct matched key-points and ASJs in the clean background, the structure of the scene can be represented by ASJs with their branches while plotted key-points are hard to understand without their input images. As shown in Fig. 9, the matched ASJs represents the geometric information with small number of ASJs (12 for Fig. 9 (a), 50 for Fig. 9 (b) and 65 for Fig. 9(c) while matched ASIFT key-points show confused results even though the amount of matches are much more than ASJs. Some example of match results are shown in Fig. 10.

6.3.2 Matching results for line-segments matching

We evaluate the matched line-segments with state-of-the-art approaches LPI [14] and LJL [15] for the comparison in a more strict rule that compare endpoints of corresponding line-segments instead of their line equation. For the example image pairs shown in Fig. 8, our proposed method is better than existing methods in considerable advantage for most cases. Some matched results for line segments are shown in Fig. 11 and Fig. 12. The number of correct matched line-segments are also comparable with other approaches. Besides of the matching accuracy, the result shown in Fig. 12 for our method cover the scene more complete.

Different from the approaches LPI [14] and LJL [15], our approach performs better while not using any pre-estimated geometric information. As shown the Tab. 3, we will find that key-point driven approach for line segment matching is possible to be failed because of the erroneous estimated geometric relationship. Observing the failed case reported in Tab. 3, the image pair in Fig. 12 is dominant by repeated texture and severe viewpoint change which are challenging for key-point matching. In such scenario, the induced epipolar geometry might be unreliable and therefore produce poor line segment matching results. On the other hand, because our approach performs well in junction matching, we can also use the junction correspondences to refine the line segment matching result.

7 Conclusion

In this paper, we proposed a novel junction detector ASJ which exploits the anisotropy of junctions via estimating the endpoints (length) of branches for isotropic scale junctions for indoor images which are dominanted by junctions in a more global manner. We then devised an affine invariant dissimilarity measure to match these anisotropic-scale junctions across different images. We tested our method on a collected indoor images and compared its performance with several current sate-of-the-art methods. The results demonstrated that our approach establishes new state-of-the-art performance on the indoor image dataset.