Progressive-X: Efficient, Anytime, Multi-Model Fitting Algorithm

TL;DR

Progressive-X is an efficient, anytime algorithm for multi-model geometric fitting that progressively explores data, providing accurate results and the ability to stop at any time with meaningful outputs.

Contribution

It introduces a novel progressive data exploration approach with a clear termination criterion and anytime capability for multi-model fitting.

Findings

Outperforms state-of-the-art in accuracy on synthetic and real datasets.

Effectively handles homography, two-view motion, and motion segmentation.

Provides reliable results even when interrupted early.

Abstract

The Progressive-X algorithm, Prog-X in short, is proposed for geometric multi-model fitting. The method interleaves sampling and consolidation of the current data interpretation via repetitive hypothesis proposal, fast rejection, and integration of the new hypothesis into the kept instance set by labeling energy minimization. Due to exploring the data progressively, the method has several beneficial properties compared with the state-of-the-art. First, a clear criterion, adopted from RANSAC, controls the termination and stops the algorithm when the probability of finding a new model with a reasonable number of inliers falls below a threshold. Second, Prog-X is an any-time algorithm. Thus, whenever is interrupted, e.g. due to a time limit, the returned instances cover real and, likely, the most dominant ones. The method is superior to the state-of-the-art in terms of accuracy in both synthetic experiments and on publicly available real-world datasets for homography, two-view motion, and motion segmentation.

Figures

Tables

Table 1. Table 1 : Fitting of multiple 2D lines to synthetic data. The worst results over five runs on the dataset of  [ 25 ] are shown. The reported properties are the number of false negative (# fn) and false positive instances (# fp). Fixed parameters were used for all methods. The results are visualized in Fig.  4 .

	\cellcolorblack!5 Progressive-X		Multi-X		PEARL		RPA		RansaCov		T-Linkage
	\cellcolorblack!5# fn	\cellcolorblack!5# fp	# fn	# fp	# fn	# fp	# fn	# fp	# fn	# fp	# fn	# fp
stair4	\cellcolorblack!51	\cellcolorblack!50	1	4	2	5	2	2	4	4	4	2
star5	\cellcolorblack!50	\cellcolorblack!50	0	0	0	0	0	0	0	0	2	0
star11	\cellcolorblack!50	\cellcolorblack!50	0	2	0	3	5	5	1	1	9	0
all	\cellcolorblack!51	\cellcolorblack!50	1	6	2	8	7	7	5	5	15	2

Table 2. Table 2 : Parameters of Prog-X used for each problem (rows). The RANSAC confidence ( μ 𝜇 \mu ); minimal Jaccard-distance for passing the instance validation ( ϵ T subscript italic-ϵ T \epsilon_{\text{T}} ); inlier-outlier threshold ( ϵ italic-ϵ \epsilon ); label cost ( w l subscript 𝑤 𝑙 w_{l} ); and the weight of the spatial coherence term ( w s subscript 𝑤 𝑠 w_{s} ) are reported.

\cellcolorblack!10	\cellcolorblack!10$\mu$	\cellcolorblack!10${ϵ}_{\text{S}}$	\cellcolorblack!10$w_s$	\cellcolorblack!10$ϵ$	\cellcolorblack!10$w_l$
Two-view motions	0.95	0.1	0.15	1.6	43.0
Homographies				2.9	112.5
Motions				$5\ast {10}^{-4}$	$2.25\ast {10}^{-6}$
Planes, cylinders				$0.5$	$0.035$

Table 3. Table 3 : Average misclassification errors (in %; 5 runs on each scene), their standard deviations and the processing times (in secs) for each problem: two-view motion fitting on the AdelaideRMF motion dataset (2nd–4th cols), homography estimation on the AdelaideRMF homography dataset (5th–7th), motion fitting on the Hopkins dataset (8th–10th), and simultaneous plane and cylinder fitting on the Multi-X dataset (11th–13th). Fixed parameters were used for all methods (see Table  2 for Prog-X). Detailed results are in Fig.  7 .

\cellcolorblack!10	\cellcolorblack!10Two-view motions			\cellcolorblack!10Homographies			\cellcolorblack!10Motions			\cellcolorblack!10Planes and cylinders
	18 scenes			19 scenes			155 scenes			8 scenes
	avg.	std.	time	avg.	std.	time	avg.	std.	time	avg.	std.	time
\cellcolorblack!5 Prog-X[12]	\cellcolorblack!510.73	\cellcolorblack!518.73	\cellcolorblack!514.38	\cellcolorblack!516.86	\cellcolorblack!515.91	\cellcolorblack!5111.03	\cellcolorblack!518.41	\cellcolorblack!510.29	\cellcolorblack!50.02	\cellcolorblack!533.69	\cellcolorblack!518.26	\cellcolorblack!51 0009.36
Multi-X1[3]	17.13	12.23	11.52	18.71	18.13	110.27	12.96	19.60	0.95	35.67	11.94	11 407.39
PEARL [15]	29.54	14.80	14.94	15.14	16.75	112.61	14.25	23.23	3.30	43.89	11.99	15 142.39
RPA [18]	17.11	11.08	10.24	23.54	13.42	622.87	19.16	11.26	4.92	55.47	11.71	10 459.98
RansaCov [19]	55.61	12.42	12.33	66.88	18.44	117.69	11.13	18.00	2.04	46.65	10.31	17 914.00
T-linkage [17]	46.67	15.60	12.69	54.79	22.17	157.84	27.24	15.57	0.95	45.33	15.37	18 423.31

Equations

$${\hat{Q}}_{\text{R}}(h,\mathcal{P},ϵ)=\sum _{\mathbf{p}\in \mathcal{P}}[\varphi (h,\mathbf{p})<ϵ\wedge \varphi (h_{\cup },\mathbf{p})\ge ϵ].$$

$$Q_{\text{M}}(h,\mathcal{P},ϵ)=|\mathcal{P}|-\sum _{\mathbf{p}\in \mathcal{P}}\min \left(1,\frac{\varphi (h,\mathbf{p}{)}^2}{\gamma (ϵ{)}^2}\right),\gamma (ϵ)=\frac{3}{2}ϵ.$$

$$\begin{array}{r}{\hat{Q}}_{\text{M}}(h,\mathcal{P},ϵ)=|\mathcal{P}|-\\ \sum _{\mathbf{p}\in \mathcal{P}}\min \left(1,\max \left(\frac{\varphi (h,\mathbf{p}{)}^2}{\gamma (ϵ{)}^2},1-\frac{\varphi (h_{\cup },\mathbf{p}{)}^2}{\gamma (ϵ{)}^2}\right)\right).\end{array}$$

$$\text{J}(h,h_{\cup })=\frac{|{\mathbf{P}}_h\cap {\mathbf{P}}_{h_{\cup }}|}{|{\mathbf{P}}_h\cup {\mathbf{P}}_{h_{\cup }}|}>{ϵ}_{\text{S}},$$

$$1-\mu \le {\left(1-{\left(|\mathcal{I}|/|\mathcal{P}|\right)}^m\right)}^k,$$

Full text

1 Introduction

The multi-class multi-model fitting problem is to interpret a set of input points as the mixture of noisy observations originating from multiple model instances which are not necessarily of the same class. Examples of this problem are the estimation of $k$ planes and $l$ spheres in a 3D point cloud; multiple line segments and circles in 2D edge maps; a number of homographies or fundamental matrices from point correspondences; or multiple motions in videos. Robustness is achieved by considering the assignment to one or multiple outlier classes.

Multi-model fitting has been studied since the early sixties. The Hough-transform [13, 14] perhaps is the first popular method for finding multiple instances of a single class [10, 20, 23, 32]. The RANSAC [9] algorithm was extended as well to deal with multiple instances. Sequential RANSAC [28, 16] detects instances one after another by searching in the point set from which the inliers of the detected instances have been removed. The greedy approach makes RANSAC a powerful tool for finding a single instance, is a drawback in multi-instance setting. Points are assigned not to the best but to the first instance, typically the one with the largest support, for which they cannot be deemed outliers. MultiRANSAC [34] forms compound hypotheses about $n$ instances. Besides requiring the number $n$ of the instances to be known a priori, the processing time is high, since samples of size $n$ times $m$ in are drawn in each iteration, where $m$ is the number of points required for estimating a model instance and finding an all-inlier simple is rare. Nevertheless, RANSAC-based approaches have a desirable property of interleaving the hypothesis generation and verification steps. Moreover, they have a justifiable termination criterion based on the inlier-outlier ratio in the data which provides a probabilistic guarantee of finding the best instance.

Most recent approaches for multi-model fitting [31, 15, 21, 17, 18, 29, 19, 3, 1] follow a two-step procedure, first, generating a number of instances using RANSAC-like hypothesis generation. Second, a subset of the generated hypotheses is selected interpreting the input data points the most. This selection is done in various ways. For instance, a popular group of methods [15, 21, 3, 1] optimizes point-to-instance assignments by energy minimization using graph labeling techniques [4]. The energy originates from point-to-instance residuals, label costs [7], geometric priors interacting among the models [21], and from the spatial coherence of the points. Another group of methods uses preference analysis based on the distribution of the residuals of data points. [33, 17, 18, 19]. Also, there are techniques [29, 30] approaching the problem as a hyper-graph partitioning where the instances are vertices, and the points are hyper-edges.

The common part of these algorithms is the initialization step when model instances are generated blindly, having no information about the data. As a consequence, it must be decided by the user whether to consider the worst case scenario and, thus, generate an unnecessarily high number of instances; or to use some rule of thumb, e.g. to generate twice the point number hypotheses. In practice, this is what is usually done. It, however, offers no guarantees for covering all the desired instances. The first method recognizing this problem was Multi-X [3]. It added a new step to the optimization procedure to reduce the number of instances by replacing sets of labels by the corresponding density modes in the model parameter domain. Even though this step allows the generation of more initial instances than before without high computational demand, it still does not provide guarantees, especially, in case of low inlier ratio.

The main contribution of the proposed Prog-X is its any-time nature which originates from interleaving sampling and consolidation of the current data interpretation. This is done via repeated instance proposal minding the already proposed ones; fast rejection of redundant hypotheses; and integration of the new hypothesis into the kept instance set by energy minimization (see Fig. 1 for the basic concept). Moreover, Prog-X adopts the probabilistic guaranties of RANSAC by progressively exploring the data. We use a general instance-to-instance metric based on set overlaps which can be efficiently estimated by the min-hash method [6] and modify the model quality function of RANSAC considering the existence of multiple model instances. The method is tested both in synthetic experiments and on publicly available real-world datasets. It is superior to the state-of-the-art in terms of accuracy for homography, two-view motion, motion clustering, and simultaneous plane and cylinder fitting.

2 Terminology and notation

In this section, we discuss the most important concepts used in this paper. For the sake of generality, we consider multi-class multi-model fitting, thus, aiming to find multiple model instances not necessarily of the same model class. We adopt the notation from [3].

Multi-class multi-instance model fitting. First, we show the problem through examples. A simple problem is finding a pair of line instances $h_{1},h_{2}\in\mathcal{H}_{l}$ interpreting a set of 2D points $\mathcal{P}\subseteq\mathbb{R}^{2}$. Line class $\mathcal{H}_{l}$ is the space of lines $\mathcal{H}_{l}=\{(\theta_{l},\phi_{l},\tau_{l}),\theta_{l}=[\alpha\quad c]^{\text{T}}\}$ equipped with a distance function $\phi_{l}(\theta_{l},p)=|\cos(\alpha)x+\sin(\alpha)y+c|\;$ ($p=[x\quad y]^{\text{T}}\in\mathcal{P}$) and a function $\tau_{l}(p_{1},...,p_{m_{l}})=\theta_{l}$ for estimating $\theta_{l}$ from $m_{l}\in\mathbb{N}$ data points. Multi-line fitting is the problem of finding multiple line instances $\{h_{1},h_{2},...\}\subseteq\mathcal{H}_{l}$, while the multi-class case is extracting a subset $\mathcal{H}\subseteq\mathcal{H}_{\forall}$, where $\mathcal{H}_{\forall}=\mathcal{H}_{l}\cup\mathcal{H}_{c}\cup\mathcal{H}_{.}\cup\cdots$. The set $\mathcal{H}_{\forall}$ is the space of all classes including that of lines and circles. Also, the formulation includes the outlier class $\mathcal{H}_{o}=\{(\theta_{o},\phi_{o},\tau_{o}),\theta_{o}=\varnothing\}$ where each instance has a constant distance to all points $\phi_{o}(\theta_{o},p)=k$, $k~{}\in~{}\mathbb{R}^{+}$ and $\tau_{o}()=\varnothing$. The objective of multi-instance multi-class model fitting is to determine a set of instances $\{h_{i}\}_{i=1}^{n_{i}}\subseteq\mathcal{H}_{\forall}$ and labeling $L\in\mathcal{P}\rightarrow\mathcal{H}$ assigning each point $p\in\mathcal{P}$ to an instance from $\{h_{i}\}_{i=1}^{n_{i}}$ minimizing energy $E$.

3 Progressive multi-model fitting

In this section, a new pipeline is proposed for multi-model fitting. Before describing each step in depth, a few definitions are discussed.

Definition 3.1**.**

A putative model instance $h_{p}$ is temporary, generated by the proposal engine, not activated to take part in the optimization procedure.

Definition 3.2**.**

An active model instance $h_{a}$ is an instance whose parameters and support are updated in the optimization procedure.

Definition 3.3**.**

Compound model instance. Given a set of activate instances $\{h_{a,i}\}_{i=1}^{n_{a}}$, where $n_{a}$ is the cardinality of the set, the compound model instance $h_{\cup}\in\mathcal{H}_{\forall}$ is defined as the union of the distance fields each generated by an individual active instance $h_{a,i}$ ($i\in[1,n_{a}]$). The distance of a point $\mathbf{p}$ from $h_{\cup}$ is $\phi(h_{\cup},\mathbf{p})=\min_{i=1}^{n_{a}}\phi(h_{a,i},\mathbf{p})$.

Examples of compound instances are shown in Fig. 3. The color codes the distance (blue – close, red – far). The left plot shows the union of the distance fields generated by two lines. The right one shows that of a circle and a line.

Definition 3.4**.**

Proposal engine $\Sigma:h_{\cup}\times\mathcal{P}^{*}\to\mathcal{H}_{\forall}^{*}$ is a function generating a putative instance from the data, using the compound model. Operator ∗ denotes the power set.

Therefore, function $\Sigma$ gets the compound model and the set of points and outputs one or multiple proposals.

Definition 3.5**.**

Multi-instance optimization procedure $\Theta:\{h_{a,i}\}_{i=1}^{n_{a}}\times\{h_{p}\}_{i=1}^{n_{p}}\times\mathcal{P}^{*}\to\{\widehat{h}_{a,i}\}_{i=1}^{\widehat{n}_{a}}\times L$ is a function getting the active, the putative instances and the data as input. It returns a set of active instances and a labeling $L:\mathcal{P}^{*}\to h_{a}^{*}$ which assigns each point to a single active instance.

Function $\Theta$ gets the active instances $\{h_{a,i}\}_{i=1}^{n_{a}}$, set of proposals $\{h_{p}\}_{i=1}^{n_{p}}$ and input points. It returns the optimized active instances and the labeling. Putative instances can be activated or rejected. Also, activate instances may be deactivated and removed here. The set of active instances always contain an instance of the outlier class.

3.1 Proposal engine

The proposal engine proposes a yet unseen instance in every iteration. For the engine, we choose a recent variant of RANSAC [9]: Graph-Cut RANSAC [2] since it is state-of-the-art and its implementation is publicly available111https://github.com/danini/graph-cut-ransac. Due to assuming local structures, similarly as in [3], we choose NAPSAC [22] to be the sampler inside GC-RANSAC. The main objective of the proposal engine is to propose unseen instances, i.e. the ones which are possibly not among to active ones in $\{h_{a,i}\}_{i=1}^{n_{a}}$. A straightforward choice for achieving this goal is to prefer instances having a reasonable number of points not shared with the compound instance $h_{\cup}$. Therefore, we propose a new quality function $\widehat{Q}:\mathcal{H}_{\forall}\times\mathcal{H}_{\forall}^{*}\times\mathcal{P}^{*}\times\mathbb{R}\to\mathbb{R}$, for RANSAC, measuring the score of a model instance originating from $\mathcal{H}_{\forall}$ w.r.t. the compound instance (from $\mathcal{H}_{\forall}^{*}$), data and a manually set threshold. For the sake of easier understanding, we start from one of the simplest quality functions, i.e. the inlier counting of RANSAC. It is as follows: $Q_{\text{R}}(h,\mathcal{P},\epsilon)=\sum_{\mathbf{p}\in\mathcal{P}}[\phi(h,\mathbf{p})<\epsilon],$ where $[.]$ is the Iverson bracket which is equal to one if the condition inside holds and zero otherwise. Based on $Q_{\text{R}}$, the modified quality function which takes $h_{\cup}$ into account and, thus, does not count the shared points is the following:

[TABLE]

The condition inside holds if and only if the distance of point $\mathbf{p}$ from instance $h$ is smaller than $\epsilon$ and, at the same time, its distance from the compound model $h_{\cup}$ is greater than $\epsilon$. Therefore, $\widehat{Q}_{\text{R}}$ counts the points which are not inliers of the compound instance but inliers of the new one.

It nevertheless turned out that, in practice, the truncated quality function of MSAC [26] is superior to the inlier counting of RANSAC in terms of accuracy and sensitivity of the user-defined threshold. It is as follows:

[TABLE]

Considering the previously described objective, $Q_{\text{M}}$ is modified as follows to reduce the score of the points which are shared with the compound model:

[TABLE]

Consequently, for point $\mathbf{p}$, the implied score is zero (i) if $\mathbf{p}$ is close to both the proposal and the compound instances, and (ii) if it is far from the proposal.

Summarizing this section, we apply GC-RANSAC as a proposal engine with NAPSAC sampling and quality function $\widehat{Q}_{\text{M}}$ to propose instances one-by-one.

3.2 Proposal validation

The validation is an intermediate step between the proposal and the optimization to decide if an instance should be involved in the optimization. To do so, an instance-to-instance distance has to be defined measuring the similarity of the proposed and compound instances. If the distance is low, the proposal is likely to be an already seen one and, thus, is unnecessary to optimize. In [3], the instances are represented by problem-specific sequences of points and This approach, leads to the question of how to represent models by points. In general, the answer is not trivial and the representation affects the outcome significantly. There, is a straightforward solution for representing instances by point sets. In [25], the models are described by their preference sets, and the similarity of two instances is defined via their Jaccard score. The preference set of instance $h$ is $\mathbf{P}_{h}\in\{0,1\}^{|\mathcal{P}|}$, where $\mathbf{P}_{h,j}$ is one if the $j$th point is an inlier of $h$, otherwise zero ($j\in[1,|\mathcal{P}|]$). The proposed criterion of accepting a putative instance is

[TABLE]

where J holds if the Jaccard similarity is higher than a manually set threshold $\epsilon_{\text{S}}\in[0,1]$ and J is false otherwise. We choose Jaccard similarity instead of Tanimoto distance [24, 17] since representing the instances by sets offers a straightforward way of speeding up the procedure.

Computing the Jaccard similarity in case of having thousands of points is a computationally demanding operation. Luckily, we are mostly interested in recognizing if the overlap of the instances is significant. The min-hash algorithm [6] is a straightforward choice for making the processing time of the similarity calculation fast and independent on the number of points. Therefore, the validation step runs in constant time.

3.3 Multi-instance optimization

The objective of this step is to optimize the set of active model instances whenever a new putative one comes and to decide if this new instance shall be activated or should be rejected. Due to aiming at the most general case, i.e. having multiple classes, there are just a few algorithms [15, 7, 21, 3] which can approach the problem without requiring non-trivial modifications. These algorithms are based on labeling energy-minimization. In general, the major issue of labeling algorithms is their computational complexity, especially, in the case of large label space. In our case, due to proposing instances one-by-one, the label space is always kept small and, therefore, the time spent on the labeling is not significant.

Multi-X [3] could be a justifiable choice. However, in our case, it is simplified to the PEARL algorithm [15, 8] since its major contribution is a move in the label space replacing a label set with the corresponding density mode. When having just a few labels, this move is not needed. Thus, we choose PEARL as the optimization procedure.222https://github.com/nsubtil/gco-v3.0

3.4 Termination criterion

In this section, we propose a criterion to determine when to stop with proposing new instances. The adaptive termination criterion of RANSAC is based on

[TABLE]

where $\mu$ is the required confidence in the results typically set to $0.95$ or $0.99$; $k$ is the number of iterations; $m$ is the size of the minimal sample; $|\mathcal{I}|$ and $|\mathcal{P}|$ are the number of inliers and points, respectively. In RANSAC, Eq. 5 is formulated to determine $k$, i.e. the number of iterations required, using the current inlier ratio. We, instead, formulate Eq. 5 to have an estimate of the maximum number of inliers independent on the compound model as follows: $(|\mathcal{P}|-|h_{\cup}|)\sqrt[m]{1-\sqrt[k]{1-\mu}}\geq|\mathcal{I}|$ where $|h_{\cup}|$ is the inlier number of the compound model. Therefore, $\overline{I}(\mathcal{P},h_{\cup},m,k,\mu)=(|\mathcal{P}|-|h_{\cup}|)\sqrt[m]{1-\sqrt[k]{1-\mu}}$ is an upper limit for the inlier number of a not yet found instance with confidence $\mu$ in the $k$th iteration.

It can be easily seen that to distinguish an instance, at least $m+1$ points have to support it, where $m$ is the size of a minimal sample. Therefore, the algorithm updates $\overline{I}$ in every iteration and terminates if $\overline{I}<m+1$. This constraint guaranties that when the algorithm terminates, the probability of having an unseen model with at least $m+1$ inliers is smaller than $1-\mu$.

Note that in practice, it is more convenient to set this limit on the basis of the optimization. For instance, if the optimization does not accept instances having fewer than $20$ inliers, it does not make sense to propose ones with fewer.

4 Experimental results

In this section, we evaluate the proposed Prog-X method on various computer vision problems that are: 2D line, homography, two-view motion, motion estimation, and simultaneous plane and cylinder fitting. The reported results are the averages over five runs and obtained by using fixed parameters. See Table 2 for the parameters of Prog-X.

4.1 Synthesized tests

In this section Prog-X is tested in a synthetic environment. We chose line fitting to 2D points and downloaded the dataset used in [25]. It consists of three scenes: stair4, star5 and star11 (see Fig. 4(a)). The outlier ratio in all test cases is $0.5$ thus having equal number of inliers and outliers.

Comparison of multi-model fitting algorithms. We compare Prog-X, Multi-X333https://github.com/danini/multi-x, [3] RansaCov [19], RPA [18], T-Linkage444http://www.diegm.uniud.it/fusiello/demo/ [17], and PEARL [5] on line fitting in this sections. All methods were applied five times to all scenes with fixed parameters. For T-Linkage, RPA and RansaCov, we used the parameters which the authors proposed. We tuned Prog-X, Multi-X, and PEARL to achieve the most accurate average results.

The worst results in five runs are visualized in Fig. 4. Plot (a) shows the ground truth lines and point-to-line assignments. The points of each cluster are drawn by color. The number of false negative, i.e. a line which is not found, and false positive, i.e. a found line which is not in the ground truth set, instances are reported in Table 1. It can be seen, that the proposed method leads to the most accurate results. It finds all but one lines and does not return false positives.

Any-time property. To demonstrate the any-time property of Prog-X, we applied the methods which minimize an energy function iteration-by-iteration, i.e. Prog-X, PEARL [5] and Multi-X [3], to the star11 scene and reported their states in each iteration. All methods were applied five times and, for each, the run with the worst outcome was selected.

Fig. 5 plots the evolution of the properties iteration-by-iteration. In (a), the numbers of stored instances (vertical axis) are plotted as the function of the iteration number (horizontal). It can be seen that PEARL and Multi-X have an unnecessarily high number of instances stored in the first iteration. Then, from the second one, it drops significantly and remains almost constant. For Prog-X, the number of instances increases by one in each iteration. The ground truth number was $11$ in this scene. In (b), it is simulated what would happen if a method is stopped before it terminates. In this case, all methods have a set of instances stored. In the plot, the ratio (vertical axis) of the number of desired instances kept and the number of all stored instances is shown as the function of the iteration number (horizontal). A desired instance is one which overlaps with a ground truth one. A ground truth instance can be covered only by one instance. The number of desired instances is, thus, at most the number of ground truth models in the data. It can be seen that PEARL and Multi-X are significantly worse than Prog-X in the first iteration since the number of stored instances is far bigger than the number of desired ones. Even if all the ground truth ones are covered, there are many false positives and, thus, the instances are not usable without further optimization. This ratio for Prog-X is one in all iterations and, as a consequence, it can be stopped any time and still returns solely desired instances. In (c), the processing times in milliseconds (vertical axis) are reported for each iteration (horizontal). PEARL and Multi-X spend seconds in the first iteration and, then, their processing times drop significantly. This is the expected behavior knowing that their most time-consuming operation is the $\alpha$-expansion algorithm which has to manage a fairly large label space in the first iterations. For Prog-X, the processing time is almost constant since it alternates between the proposal and optimization step with slightly increasing label space. Summarizing (a–c), PEARL and Multi-X are not any-time algorithms. To (a) and (b), they can be stopped after the first iteration without significant deterioration in the accuracy. However, to (c), the first iteration requires more processing time than the rest in total. Prog-X can be interrupted at any time; all of the stored instances cover ground truth ones.

Plot (d) shows the change of the energy (vertical axis) iteration-by-iteration (horizontal). We divided the energy in each iteration by the energy in the first one. It can be seen that Prog-X leads to a significantly higher reduction in the energy than the competitor algorithms.

Comparison with increasing outlier ratio. To evaluate, how the outlier ratio affects the outcome of multi-model fitting algorithms, we first kept solely the noisy inliers from the star11 scene. Then outliers, i.e. points uniformly distributed in the scene, were added to achieve a given outlier ratio $\nu$. Fig. 6 reports (a) the processing time (in milliseconds; vertical axis) and (b) the difference between the ground truth and the obtained instance number (vertical) as the function of the outlier ratio (horizontal). It can be seen that the proposed Prog-X leads to similar processing time as Multi-X but the returned number of instances is significantly closer to the desired one. Consequently, it is the least sensitive to the outlier ratio.

4.2 Real world problems

To evaluate the proposed method on real-world problems, we downloaded a number of publicly available datasets. The error is the misclassification error (ME), i.e. the ratio of points assigned to the wrong cluster. In Fig. 2, the first images are shown of image pairs which are some of the worst (left plots) and best (right) results of Prog-X running five times on each scene with a fixed parameters (reported in Table 2). The shown percentages are the misclassification errors. The white points are labeled as outliers, while the ones with color are assigned to an instance. It can be seen the error originates mostly from missing instances and, thus, the returned ones are usable.

Two-view motion fitting is evaluated on the AdelaideRMF motion dataset consisting of $21$ image pairs of different sizes and correspondences manually assigned to motion clusters. In this case, multiple fundamental matrices were fit. In the proposal step, the 7-point algorithm [11] was applied for fitting to a minimal sample and the normalized 8-point method [12] for the polishing steps on non-minimal samples. The average (of 5 runs with fixed parameters) errors and their standard deviations are shown in the first block of Table 3 for each method (from the $2$nd to the $4$th columns). Prog-X is superior to the competitors in all investigated properties. Detailed results for each scene are shown in the top-left plot of Fig. 7. It can be seen that Prog-X is always the most or the second most accurate method.

Homography fitting is evaluated on the AdelaideRMF homography dataset [31] used in most recent publications. AdelaideRMF consists of $19$ image pairs of different resolutions with ground truth point correspondences assigned manually to homographies. In the proposal step, the normalized 4-point algorithm [11] was used for fitting to a minimal sample and also for the polishing steps. The results are shown in the second block of Table 3 (from the $5$th to the $7$th columns). Prog-X leads to the lowest average error and standard deviation. Similar results can be seen in the top-right plot of Fig. 7 as before: Prog-X always leads to the lowest or second lowest misclassification errors.

Motion segmentation is tested on the $155$ videos of the Hopkins dataset [27]. The dataset consists of $155$ sequences divided into three categories: checkerboard, traffic, and other sequences. The trajectories are inherently corrupted by noise, but no outliers are present. Motion segmentation in videos is the retrieval of sets of points undergoing rigid motions contained in a dynamic scene captured by a moving camera. It can be considered as a subspace segmentation under the assumption of affine cameras. For affine cameras, all feature trajectories associated with a single moving object lie in a 4D linear subspace in $\mathbb{R}^{2F}$, where $F$ is the number of frames [27]. The results are shown in the third block of Table 3 (from the $8$th to the $10$th columns). Prog-X leads to the lowest average errors. Its standard deviation is the second lowest. Detailed results on the sequences of traffic are put in the bottom of Fig. 7.

Plane and cylinder fitting is evaluated on the dataset from [3]. It consists of LiDAR point clouds of traffic signs, their columns and the neighboring points. Points were manually assigned to signs (planes) and columns (cylinders). The proposal step of Prog-X alternately proposes cylinders and planes. The results are in the last three columns of Table 3. Prog-X obtains the most accurate results, but, most importantly, it is three orders of magnitude faster than the second fastest method. The reason is the large number of points in the scenes (from $1\;260$ up to $52\;445$). The processing time of Prog-X, due to being dominated by a number of RANSAC runs, depends linearly on the point number. All of the other methods depend approximately quadratically on $n$. In RPA, T-Linkage and RansaCov, the preference calculation has $\mathcal{O}(n^{2})$ complexity. In Multi-X and PEARL, if at least $n$ initial instances are generated, the label space is already too large for the inner $\alpha$-expansion to finish early.

5 Conclusion

The Prog-X algorithm is proposed for geometric multi-class multi-model fitting. The method is fast and superior to the state-of-the-art in terms of accuracy in a synthetic environment and on publicly available real-world datasets for homography, two-motion, and motion segmentation. Additionally, it is an any-time algorithm. Therefore, whenever is interrupted, e.g. due to a strict time limit, the returned instances cover real and, likely, the most dominant ones. The termination criterion, adopted from RANSAC, makes Prog-X robust to the inlier-outlier ratio.