Robust Motion Segmentation from Pairwise Matches

TL;DR

This paper introduces a novel two-step approach for motion segmentation using only pairwise matches, first segmenting image pairs independently and then combining results to achieve globally consistent segmentation, effectively handling mismatches.

Contribution

It presents a new formulation for motion segmentation from pairwise matches, including a robust method to combine independent results into a consistent global segmentation.

Findings

Effective in reducing segmentation errors

Handles large numbers of mismatches

Works well in both simulated and real experiments

Abstract

In this paper we address a classification problem that has not been considered before, namely motion segmentation given pairwise matches only. Our contribution to this unexplored task is a novel formulation of motion segmentation as a two-step process. First, motion segmentation is performed on image pairs independently. Secondly, we combine independent pairwise segmentation results in a robust way into the final globally consistent segmentation. Our approach is inspired by the success of averaging methods. We demonstrate in simulated as well as in real experiments that our method is very effective in reducing the errors in the pairwise motion segmentation and can cope with large number of mismatches.

Figures

Tables

Table 1. Table 1: Average misclassification error [ % percent \% ] for several methods on the Hopkins155 benchmark [ 47 ] . Results are copied from [ 58 ] .

	LSA [59]	GPCA [50]	ALC [36]	SSC [11]	TPV [24]	LRR [25]	T-Linkage [27]	S3C [22]	RSIM [17]	MSSC [21]	KerAdd [58]	Coreg [58]	Subset [58]	Baseline	Mode
2 Motions	4.23	4.59	2.40	1.52	1.57	1.33	0.86	1.94	0.78	0.54	0.27	0.37	0.23	2.26	1.00
3 Motions	7.02	28.66	6.69	4.40	4.98	4.98	5.78	4.92	1.77	1.84	0.66	0.75	0.58	9.04	2.67
All	4.86	10.02	3.56	2.18	2.34	1.59	1.97	2.61	1.01	0.83	0.36	0.46	0.31	3.79	1.37

Table 2. Table 2: Average and median misclassification error [ % percent \% ] for several methods on the Hopkins12 benchmark [ 52 ] . Results for different variants of ALC and SSC are taken from [ 17 ] whereas results for the remaining methods are copied from the respective papers.

	PF [52]	PF+ALC [36]	RPCA+ALC [36]	${\ell }_1$+ALC [36]	SSC-R [11]	SSC-O [11]	RSIM [17]	KerAdd [58]	Coreg [58]	Subset [58]	Baseline	Mode
Mean	14.94	10.81	13.78	1.28	3.82	8.78	0.61	0.11	0.06	0.06	7.45	4.33
Median	9.31	7.85	8.27	1.07	0.31	4.80	0.61	0.00	0.00	0.00	2.16	0.38

Table 3. Table 3: Misclassification error [ % percent \% ] and classified points [ % percent \% ] for several methods on our dataset. The number of motions d 𝑑 d , the number of images n 𝑛 n , and the total number of image points p 𝑝 p are also reported for each sequence.

				Mode		Baseline		StableSfM + Subset [58]		QuichMatch + Subset [58]		StableSfM + RSIM [17]		QuichMatch + RSIM [17]
Dataset	$d$	$n$	$p$	Error	Classified	Error	Classified	Error	Classified	Error	Classified	Error	Classified	Error	Classified
Penguin	2	6	5865	0.76	69.17	0.95	33.95	32.27	99.59	41.05	70.11	41.50	99.59	41.54	70.11
Flowers	2	6	7743	1.23	73.65	2.84	32.70	8.55	99.50	8.59	72.59	16.65	99,50	14.20	72.59
Pencils	2	6	2982	3.80	65.33	2.30	30.65	41.46	99.56	40.88	66.36	23.07	99.56	23.45	66.36
Bag	2	7	6114	1.52	57.95	1.54	26.56	14.22	99.69	15.67	65.85	34.55	99.69	39.92	65.85
Bears	3	10	15888	4.82	73.65	2.72	29.80	38.13	99.58	35.21	63.12	49.48	99.58	53.80	63.12

Table 4. Table 4: Misclassification error [ % percent \% ] and classified points [ % percent \% ] for several methods on outdoor scenes. The number of motions d 𝑑 d , the number of images n 𝑛 n , and the total number of image points p 𝑝 p are also reported for each sequence.

				Mode		Baseline		StableSfM + Subset [58]
Dataset	$d$	$n$	$p$	Error	Classified	Error	Classified	Error	Classified
helicopter [10]	2	10	17139	2.01	80.82	0.78	45.93	16.81	99.52
boat [24]	2	10	21183	–	87.34	–	56.31	–	99.62
cars7 [47]	2	21	16602	–	92.27	–	57.38	–	99.66
cars8 [47]	2	19	13438	–	93.12	–	50.53	–	99.61

Equations

$${\hat{\mathbf{s}}}_i^j\in \{\text{NaN},0,1,\dots ,d{\}}^{p_i}$$

$${\hat{\mathbf{s}}}_j^i\in \{\text{NaN},0,1,\dots ,d{\}}^{p_j}$$

$${\mathcal{B}}_i=\{{\hat{\mathbf{s}}}_i^k\text{ s.t. }k=1,\dots ,n,k\ne i\}.$$

$${\mathbf{s}}_i(r)=\text{mode }\{{\hat{\mathbf{s}}}_i^k(r)\text{ s.t. }{\hat{\mathbf{s}}}_i^k\in {\mathcal{B}}_i,{\hat{\mathbf{s}}}_i^k(r)\ne \text{NaN}\}$$

Full text

1 Introduction

Motion segmentation is an essential task in many applications in Computer Vision and Robotics, such as surveillance [18], action recognition [56] and scene understanding [12]. The classic way to state the problem is the following: given a set of feature points that are tracked through a sequence of images, the goal is to cluster those trajectories according to the different motions they belong to. It is assumed that the scene contains multiple objects that are moving rigidly and independently in 3-space. There is a plenty of available techniques to accomplish such task, as detailed in Sec. 1.1. Among them, the methods developed in [17, 21, 58] achieve a very low misclassification error on the Hopkins155 benchmark [47], which is a well established dataset to test the performance of motion segmentation. However, the tracks available in the dataset are not realistic at all since they were filtered with the aid of manual operations.

In this paper we take motion segmentation one step further by considering more difficult/realistic assumptions, namely we assume that pairwise matches (e.g. those computed from SIFT keypoints [26]) are available only, and we address the task of classifying image points (instead of tracks), as shown in Fig. 1. This problem has not been considered before but it has great practical relevance since it does not require to compute tracks in advance, which is a challenging task in the presence of multiple moving objects.

More precisely, we formulate motion segmentation as a two-step process:

segmentation of corresponding points is performed on each image pair in isolation; 2. 2.

segmentation of image points is computed without relying on the feature locations, using only the classification of matching points derived in Step 1.

Our new formulation is detailed in Sec. 2. Regarding Step 2, we develop a simple scheme that classifies each point based on the frequencies of labels of that point in different image pairs, which is reported in Sec. 3. The resulting method is a general framework that can be combined with any algorithm able to perform motion segmentation in two images.

The idea of combining results from individual image pairs was also present in [24], where all the pairs were considered, and in [21, 58], where only pairs of consecutive frames were used. These techniques, however, are different from our approach since they do not completely perform segmentation of image pairs but they rely on partial results only (i.e. correlation of corresponding points). Such results are used to build an affinity matrix that encodes the similarity between different tracks, to which spectral clustering [54] (or its multi-view variations [6, 20, 55]) is applied. As a consequence, they perform segmentation of tracks, whereas our method classifies image points. A related approach [16] considers the scenario where correspondences are unknown and uses the Alternating Direction Method of Multipliers (ADMM) [3] to jointly perform motion segmentation and tracking, where sequences with at most 200 trajectories are analyzed only due to algorithmic complexity.

Experiments on both synthetic and real data were performed to validate our approach. Robust Preference Analysis (RPA) [28] was used in Step 1. A new dataset was also created, consisting of five sequences of moving objects in an indoor environment, where SIFT keypoints [26] were extracted and manually labelled to get ground-truth segmentation. Results are reported in Sec. 4, where it is shown that: our method is comparable or better than most traditional (track-based) solutions on Hopkins155 [47]; it outperforms the methods developed in [17, 58] on synthetic/real datasets with mismatches; it is very effective in reducing the errors in the pairwise segmentations; it can be profitably used to segment SIFT keypoints in a collection of images.

Our two-step formulation of motion segmentation is inspired by the success gained by synchronization or averaging methods (e.g. [42, 46, 14, 1, 4]) that formulate other computer vision problems (e.g. structure from motion and multi-view registration) in an analogous manner. For instance, multi-view registration – where the task is to bring multiple scans into alignment – can be addressed by first computing rigid transformations between each pair of scans in isolation, and then globally optimizing these transformations without considering points. In particular, our method – which computes the segmentation of one image at a time (as explained in Sec. 3) – presents similarities with [46, 14], which estimate the transformation of one camera/scan at a time.

1.1 Related Work

Motion segmentation lies at the intersection of several computer vision problems, including subspace separation, multiple model fitting and multibody structure from motion.

The goal of subspace separation is to cluster high-dimensional data drawn from multiple low-dimensional subspaces. Existing solutions include Generalized Principal Component Analysis (GPCA) [50], Local Subspace Affinity (LSA) [59], Power Factorization (PF) [52], Agglomerative Lossy Compression (ALC) [36], Low-Rank Representation (LRR) [25], Sparse Subspace Clustering (SSC) [11], Structured Sparse Subspace Clustering (S3C) [22], and Robust Shape Interaction Matrix (RSIM) [17]. Motion segmentation can be cast as subspace separation since – under the affine camera model – the point trajectories lie in the union of $d$ subspaces in $\mathbb{R}^{2n}$ of dimension at most 4, where $d$ denotes the number of motions and $n$ denotes the number of images. Subspace separation techniques can also be used to solve motion segmentation in two images under the perspective camera model, since corresponding points undergoing the same motion – after a proper rearrangement of coordinates – belong to a subspace of $\mathbb{R}^{9}$ of dimension at most 8, as observed in [24].

The goal of multiple model fitting is to estimate multiple models (e.g. geometric primitives) that fit data corrupted by outliers and noise, without knowing which model each point belongs to. Some methods follow a consensus-based approach, namely they focus on the estimation part of the problem, with the aim of finding models that describe as many points as possible. The Hough transform [57], Sequential RANSAC [53], Multi-RANSAC [62] and Random Sample Coverage (RansaCov) [29] belong to this category. Other techniques follow a preference-based approach, namely they concentrate on the segmentation side of the problem, from which model estimation follows. Solutions of this type include Residual Histogram Analysis (RHA) [61], J-Linkage [44], Kernel Optimization [5], T-linkage [27], Random Cluster Model (RCM) [34] and Robust Preference Analysis (RPA) [28]. The problem of fitting multiple models can also be expressed in terms of energy minimization [8, 9], as done by PEARL (Propose Expand and Re-estimate Labels) [15] and Multi-X [2]. Model fitting techniques can be exploited to solve motion segmentation under the affine camera model, by fitting multiple subspaces to feature trajectories in an image sequence, similarly to subspace separation methods. They can also be used to solve motion segmentation in two images under the perspective camera model, by fitting multiple fundamental matrices to corresponding points in an image pair.

The goal of multibody structure from motion is to simultaneously estimate the motion between each object and the camera as well as the 3D structure of each object, given a set of images of a dynamic scene. This problem can be seen as the generalization of structure from motion [32] to the dynamic case, where motion segmentation has to be solved in addition to 3D reconstruction. Geometric solutions are available for two images [51] and three images [49]. Other techniques follow a statistical approach [45, 35, 41, 43, 31, 38], whereas in [13, 7, 23, 60] motion segmentation and structure from motion are combined. More details can be found in survey [40].

Ad-hoc solutions to motion segmentation are also present in the literature [24, 21, 58], which are not explicitly related to the aforementioned problems. The authors of [24] formulate a joint optimization problem which builds upon the SSC algorithm, where it is required that all image pairs share a common sparsity profile. In [21] an accumulated correlation matrix is built by sampling homographies over consecutive image pairs, and spectral clustering [54] is applied to get the sought segmentation. Such approach is generalized in [58] where multiple models (affine, fundamental and homography) are combined to get an improved segmentation. Different approaches are analyzed to reach such task, namely Kernel Addition (KerAdd) [6], Co-Regularization (Coreg) [20] and Subset Constrained Clustering (Subset) [55]. Motion segmentation is also addressed in [39, 37], where existing algorithms are customized for specific scenarios and acquisition platforms.

2 Problem Formulation

Let $n$ denote the number of images and let $d$ denote the number of motions. Suppose that a number $p_{i}$ of points are found in image $i$ using a feature extraction algorithm, so that the total amount of points over all the images is given by $p=\sum_{i=1}^{n}p_{i}$. Let $\mathbf{s}_{i}\in\{0,1,\dots,d\}^{p_{i}}$ denote the labels of points in image $i$, which identify the membership to a specific motion. The meaning of the zero label, which essentially represents outliers, will be clarified in Sec. 3.3. The vector $\mathbf{s}_{i}$ is referred to as the total segmentation of image $i$, since it represents labels of points considering a global numbering of motions. The goal here is to estimate $\mathbf{s}_{i}$ for $i=1,\dots,n$, as shown in Fig. 2. In other words, we aim at classifying image points as opposed to existing methods which segment tracks. In order to accomplish such a task, we assume that points have been matched in image pairs and that segmentation between pairs of images is available. Note that the knowledge of matches, which involve two images at a time, is a weaker assumption than the presence of tracks, which involve all the images simultaneously.

Let $\mathbf{s}_{ij}\in\{0,1,\dots,d\}^{m_{ij}}$ denote the labels of corresponding points in images $i$ and $j$, where the zero label corresponds to outliers and let $m_{ij}\leq\min\{p_{i},p_{j}\}$ denote the number of matches of the pair $(i,j)$. Vector $\mathbf{s}_{ij}$ is referred to as the partial segmentation of the pair $(i,j)$, since it represents labels of corresponding points considering a local numbering of motions, as shown in Fig. 3. Observe that each $\mathbf{s}_{ij}$ may contain some errors, which can be caused either by mismatches or by failure of the algorithm used for pairwise segmentation, and some image points may not have a label assigned in some pairs due to missing correspondences.

Thus we have to face the problem of how to assign a unique/global label to all image points such that the constraints coming from pairwise segmentation are best satisfied. In other words, the segmentation task can be reduced to the problem of estimating the total segmentations $\mathbf{s}_{1},\dots,\mathbf{s}_{n}$ starting from the knowledge of partial segmentations $\mathbf{s}_{ij}$ with $i,j=1,\dots,n$. It is worth noting that in this way the actual coordinates of image points are not used anymore after pairwise segmentation, since only labels matter for the final segmentation. Observe also that this general formulation does not assume any particular camera model or scene geometry.

3 Proposed Method

Our method (sketched in Fig. 7) takes as input the results from pairwise segmentation. It first computes the total segmentation of each image individually and then updates all these estimates in order to have a single/global numbering of motions.

3.1 Segmenting a single image

The key observation is that each partial segmentation $\mathbf{s}_{ij}\in\{0,1,\dots,d\}^{m_{ij}}$ gives rise to two vectors

[TABLE]

which contain labels of matching points in images $i$ and $j$, respectively, where NaN accounts for missing correspondences. This implies that, if we fix one image (e.g. image $i$), then several estimates are available for its total segmentation, which define a set $\mathcal{B}_{i}$

[TABLE]

However, these estimates are not absolute since they may differ by a permutation of the labels associated with each motion, as shown in Fig. 4.

In order to resolve such ambiguity, we consider a graph where each node is an element in $\mathcal{B}_{i}$ (i.e. a partial segmentation involving image $i$) and the edge between nodes $h$ and $k$ is associated with a permutation $P_{hk}$ of labels that best maps $\widehat{\mathbf{s}}_{i}^{k}$ (i.e. labels of image $i$ in the pair $(i,k)$) into $\widehat{\mathbf{s}}_{i}^{h}$ (i.e. labels of image $i$ in the pair $(i,h)$). Computing such permutation is a linear assignment problem, which can be solved using the Hungarian algorithm [19]. The task here is to compute a permutation $P_{k}$ for each node that reveals the true numbering of motions. It can be seen that this can be expressed as a permutation synchronization, that is the problem of estimating $P_{k}$ for $k=1,\dots,n$ ($k\neq i$) such that $P_{hk}=P_{h}P_{k}^{-1}$, which can be solved via eigenvalue decomposition [33].

After this step, the set in Eq. (3) contains several estimates of $\mathbf{s}_{i}$ with respect to a single numbering of motions, as shown in Fig. 5. Thus a scheme that assigns a unique label to each point in image $i$ is required, which can be regarded as the best over the set $\mathcal{B}_{i}$. A reasonable approach consists in labelling each point with the most frequent label (i.e. the mode) among all the available measures. In other words, the label of point $r$ is given by

[TABLE]

where only labels of actual correspondences are considered, with $r=1,\dots,p_{i}$. As long as the algorithm used for pairwise segmentation correctly classifies all the points in most pairs, this procedure works well, as confirmed by experiments in Sec. 4.

3.2 Segmenting multiple images

The above procedure is applied to all the images in order to estimate the sought total segmentations $\mathbf{s}_{1},\mathbf{s}_{2}\dots,\mathbf{s}_{n}$. Such estimates, however, are not absolute since each image has been treated independently from the others, and hence results may differ by a permutation of the labels associated with each motion, as shown in Fig. 6.

In order to address this issue, we consider a graph where each node corresponds to an image and the edge between images $i$ and $j$ is associated with a permutation $P_{ij}$ that best maps $\mathbf{s}_{j}$ into $\mathbf{s}_{i}$. In order to compute such permutation, we ground on pairwise segmentation, since labels of the same points are required: in order to map $\mathbf{s}_{j}$ (labels of image $j$) into $\mathbf{s}_{i}$ (labels of image $i$), we first map $\widehat{\mathbf{s}}_{i}^{j}$ (labels of image $i$ in the pair $(i,j)$) into $\mathbf{s}_{i}$, and then we map $\mathbf{s}_{j}$ into $\widehat{\mathbf{s}}_{j}^{i}$ (labels of image $j$ in the pair $(i,j)$). These are linear assignment problems [19]. Thus the task is to compute a permutation $P_{i}$ for each image that reveals the true numbering of motions such that $P_{ij}=P_{i}P_{j}^{-1}$, which can be viewed as a permutation synchronization [33]. Hence all the total segmentations are expressed with respect to the same numbering of motions, as in Fig. 2.

3.3 Dealing with outliers

When doing pairwise segmentation, it is expected that mismatched points are classified as outlier (zero label). When dealing with total segmentation, instead, the situation is different: in principle, there exists no outlier since each image point actually belongs to a motion. However, in the presence of high corruption in the input matches, one may not be able to assign a valid label to all image points. Indeed, it may happen that a point is mismatched (and hence assigned the zero label) in all the pairs, so that there is no valid information to classify it. Such points are expected to have zero label in the absolute segmentation. However, since they are not actual outliers, we will refer to them as “unclassified” or “unknown” in the experiments.

In order to deal with those points, a reasonable approach is to ignore the labels which are set to zero by pairwise segmentation and compute the mode over the remaining measures, i.e. substitute them with NaN before using Eq. (4). In this way all the image points are assigned a valid label (except those which are deemed as outlier in all the pairs), meaning that this approach tends to classify a high amount of points even in the presence of mismatches.

4 Experiments

In order to evaluate the performance of our approach – named Mode 111The Matlab code will be made available on the web. – we ran experiments on both synthetic data and real images, in addition to the real data Hopkins155 [47] and Hopkins12 [52]. For pairwise segmentations – which constitute the input to our method – we fitted multiple fundamental matrices to correspondences in each image pair using RPA [28] (code available online222\urlhttp://www.diegm.uniud.it/fusiello/demo/rpa/). Default values specified in the original paper were used for the algorithmic parameters in all the experiments.

Note that there are no direct competitors to our method, since the task of segmentation from pairwise matches has not been addressed so far. For this reason, we will focus on the comparison with a trivial solution (named the “baseline”) which takes the same input as our approach (i.e. the results from pairwise segmentation) and it is constructed as follows: first, a maximum-weight spanning tree is computed, where each node in the graph is an image and edges are weighted with the number of inliers; then, the results from pairwise segmentation are used to segment each image along the tree, where the global numbering of motions is fixed at the root and propagated to the leaves.

Similarly to most works in motion segmentation literature, we assume that the number of motions is known in advance and give this value as input to all the analysed techniques.

4.1 Hopkins Datasets

The Hopkins155 benchmark [47] contains 155 sequences of indoor and outdoor scenes with two or three motions, which are categorized into checkerboard, traffic and articulated/nonrigid sequences, and the Hopkins12 dataset [52] provides 12 additional sequences with missing data. We emphasize that these datasets provide (cleaned) tracks over multiple images, so they are not suitable for the task addressed in this paper, which is segmentation from raw pairwise matches. However, we report results on these sequences since they are widely used in segmentation literature.

In order to make a meaningful comparison with the state of the art, a scheme that assigns a unique label to each track is required, starting from labels of image points. To accomplish such a task, we use the same criterion as the one developed in Sec. 3 to label each image point given multiple measures derived from pairwise segmentation. We assign to each track the mode of the labels of points belonging to the track, and the same procedure is applied to the baseline. Performance is measured in terms of misclassification error, that is the percentage of misclassified tracks, as it is customary in motion segmentation literature. Tracks labelled as zero (if any) were counted as errors, since we know that outliers are not present in these datasets.

Results are reported in Tab. 1 and Tab. 2 where Mode is compared to several motion segmentation algorithms. Our approach clearly outperforms the baseline and it performs comparably or better than most of the state-of-the-art techniques, with a mean error of $1.37\%$ over all the sequences in Hopkins155 and a median error of $0.38\%$ over all the sequences in Hopkins12. In particular, it is noticeable that our method achieves (nearly) zero error in 139 out of 155 sequences in Hopkins155 and in 10 out of 12 sequences in Hopkins12, as shown in Fig. 8. After inspecting the solution, it was found that the remaining sequences correspond to situations where the algorithm used for pairwise segmentation (RPA) performed bad in most image pairs.

The fact that our method is not the best is not surprising since we are making much weaker assumptions (matches between image pairs instead of tracks over multiple images), i.e., we are addressing a more difficult task. Nevertheless, our method achieves good performances. In general, there is no reason to use our approach when tracks are available and one out of the best traditional methods (e.g.[17, 21, 58]) can be used. Our method is designed for the scenario where pairwise matches are available only. The next sections demonstrate the benefits of our approach for this specific task.

4.2 Simulated Data

We considered the cars1 dataset from the traffic sequences in Hopkins155, where $d=2$, $n=20$ and $p=6140$. Noise-free pairwise matches were obtained from the available tracks and synthetic errors were added to these correspondences in order to produce mismatches. More precisely, in each image pair a fraction of the correspondences – which ranged from 0 to 0.8 in our experiments – was randomly switched. This scenario resembles unordered image collections (e.g. in multibody structure from motion) where errors are ubiquitous among pairwise matches. For each configuration the test was repeated 10 times and average results were computed.

We compared Mode with the baseline, which – as our method – takes as input the results from pairwise segmentation. We also included in the comparison two traditional methods which require tracks over multiple images as input, namely RSIM333\urlhttps://github.com/panji1990/Robust-shape-interaction-matrix [17] and Subset444\urlhttps://alex-xun-xu.github.io/ProjectPage/CVPR_18/ [58], whose implementations are available online. The former provides a robust solution to subspace separation, whereas the latter can be regarded as the current state of the art in motion segmentation with mean error of $0.31\%$ on the Hopkins155 benchmark (see Tab. 1). We used two different techniques for computing tracks from pairwise matches, namely StableSfM555\urlhttp://www.maths.lth.se/matematiklth/personal/calle/sys_paper/sys_paper.html [30] and QuichMatch666\urlhttps://bitbucket.org/tronroberto/quickshiftmatching [48].

Performance was measured in terms of misclassification errors, which is defined here as the percentage of misclassified points over the total amount of classified image points. In other words, unlike in Sec. 4.1, segmentation results were evaluated considering only points with a nonzero label (i.e. points with zero label do not contribute to the error). Indeed, due to the presence of mismatches, one may not expect to give a valid label to all the image points, as observed in Sec. 3.3. We also computed the percentage of points classified by each method.

Results are reported in Fig. 9, which clearly shows the robustness to mismatches gained by our approach: it is remarkable that the error remains constant (around $0\%$) with up to $60\%$ of mismatches. Mode is significantly better than the baseline both in terms of misclassification error and percentage of classified points. The former exploits redundant measures in order to produce the final segmentation, whereas the latter uses results from a maximum-weight spanning tree only.

Concerning traditional methods, it was found by inspecting the solution that Subset and RSIM actually segment all the tracks, and unclassified data correspond to image points that were not included in any track by the algorithm used for computing tracks. Such techniques achieve a low misclassification error only when mismatches are below $10\%$ and performances degrade with increasing ratio of mismatches. Indeed, wrong correspondences propagate into the tracks making traditional motion segmentation really hard to solve. Notice that a track can even contain points of different motions, in which case errors in the output segmentation appear by assigning a unique label to the entire track. This clearly motivates the need of our method for segmentation from raw pairwise matches.

In order to give a full picture on the performance of our approach, we report in Fig. 10 the histograms of misclassification error achieved by RPA over all the image pairs, which gives an idea about how hard it is to solve the motion segmentation given results of pairwise segmentation. Indeed, RPA may fail to detect errors in the input matches and it may not correctly segment some points since it lacks theoretical guarantees, thus producing errors in the individual pairwise segmentations. As expected, the histograms shift to the right as the percentage of input mismatches increases. Let us consider the central histogram, which corresponds to 60$\%$ of mismatches: it is worth noting that, despite individual pairwise segmentations are noisy, our method achieves zero error, as shown in Fig. 9. In other words, Mode is able to successfully solve motion segmentation while reducing errors in the pairwise segmentations, thanks to the fact that it exploits redundant measures in a principled manner.

We now illustrate what happens to individual points when running our method. Figure 11 reports coloured bars representing the amount of errors for each point in a sample image. As the percentage of mismatches increases, motion segmentation gets harder to solve, since the green area reduces whereas the blue and red ones enlarge. Note that RPA [28] produces errors even in the absence of wrong correspondences, as can be appreciated in Fig. 11a. Our method classifies all the data except for a few cases where the blue bars are equal to 1, meaning that the point is labelled as outlier by RPA in all the pairs. Among the classified points, Mode provides a correct segmentation as long as the green bars are sufficiently high.

4.3 Real Data

In order to evaluate the performance of our approach on real data, we considered both indoor and outdoor images. SIFT keypoints [26] were extracted in all the images and correspondences between image pairs were established using the nearest neighbor and ratio test as in [26], using the VLFeat library777\urlhttp://www.vlfeat.org/. For each image pair $(i,j)$, we kept only those correspondences that were found both when matching image $i$ with $j$ and when matching image $j$ with $i$, and isolated features (i.e. points that are not matched in any image) were removed. No further filtering was applied.

4.3.1 Indoor scenes

Since there are no standard datasets for segmentation from pairwise matches, we created a small benchmark888The dataset will be made available on the web. consisting of five image sequences. We considered indoor scenes containing two or three motions where one object is fixed (i.e. it is a part of the background), and we acquired from 6 to 10 images of size $2922\times 2000$ with a moving camera. Fig. 12 shows a sample image from each sequence. SIFT correspondences on such images are very noisy, as shown in Fig. 13, making motion segmentation a challenging task. In the case of the Penguin sequence there is no motion between some frames, so pairwise segmentation was not performed. In the remaining sequences, RPA was applied to all the image pairs.

As in Sec. 4.2, we compared Mode with the baseline, which takes as input the results from pairwise segmentation, and we also considered two traditional methods, namely RSIM [17] and Subset [58], where StableSfM [30] and QuichMatch [48] were used to compute tracks over multiple images. In order to evaluate results quantitatively, we manually labelled points in each sequence, thus producing a ground-truth segmentation of each image, that was used to compute the misclassification error. The number of points that undergo the same motion is reported in Fig. 15, which gives an idea about the distribution of points in the scene for each sequence. Results are shown in Tab. 3, which also reports the percentage of points classified by each method.

While there are no significant differences between Mode and the baseline in terms of misclassification error, the former is superior in terms of the percentage of classified points since it exploits redundant two-frame segmentations. Both our method and the baseline – with a misclassification error lower than $5\%$ in all the sequences – are significantly better than Subset and RSIM. Traditional methods exhibit poor performances on our dataset since they do not deal with mismatches, confirming the outcome of the experiments on synthetic data.

Figures 1, 16, 17, 18, 19 and 20 visually represent the segmentation of image points obtained by several methods, which complement the quantitative evaluation provided in Tab. 3. Ground-truth segmentation is also shown. Concerning the different variants of Subset [58] and RSIM [17], which differ for the algorithm used for computing tracks, we report results for StableSfM [30] only. Indeed, there are not significative differences between StableSfM [30] and QuichMatch [48] in terms of misclassification error, but the former is better in terms of amount of classified data. Our method returns high quality (although not perfect) segmentation in all the sequences, outperforming the baseline in terms of percentage of classified points, whereas Subset and RSIM present poor performances on our benchmark.

In order to give further insights on the behavior of our technique, we report in Fig. 14 the histograms of misclassification error achieved by RPA [28] over image pairs, similarly to Fig. 10. The histograms show the effective amount of corruption in the data after performing pairwise segmentation with RPA, which is the first step of our pipeline. Note that the misclassification error exceeds $30\%$ in some image pairs from the Bears sequence. It is remarkable that our method is able to achieve a low error in this dataset (about $4.8\%$), as reported in Tab. 3. In other words, it can effectively reduce errors in the pairwise segmentations thanks to the fact that it exploits redundant measures.

We also tested the method developed in [16], which does not require pairwise matches but feature locations and descriptors only. We ran the available Matlab implementation of [16] on Pencils sequence. It did not return any solution after several hours of computation due to “out of memory” error. We conclude that it is not yet a practical approach to motion segmentation on the scenarios considered in our paper.

4.3.2 Outdoor scenes

To study a more realistic scenario, we considered four outdoor scenes, namely helicopter [10], boat [24], cars7 [47] and cars8 [47], which are shown in in Fig. 21, 22, 23 and 24. A subset of the images was chosen for each sequence in order to ensure enough motion between consecutive frames. The properties of each dataset are presented in Tab. 4, which also reports the percentage of points classified by Mode, the baseline and Subset [58] combined with StableSfM [30]. The latter provided the best results among all possible combinations of traditional segmentation methods and tracking algorithms. In the case of the helicopter sequence, a subset of the images has ground-truth pixel-wise annotation, which was used to compute the misclassification error (see Tab. 4). For the remaining sequences, no ground-truth is available, so only qualitative evaluation can be provided, which is reported in Fig. 21, 22, 23 and 24.

Results show that our solution is of good quality in all the images, outperforming the baseline in terms of amount of classified data. The poor performance of the baseline on some images gives an idea about how noisy the individual pairwise segmentations are. Our method is able to reduce such errors thanks to the fact that it exploits redundant measures. There are no significant differences between Subset and Mode in the boat sequence, which, however, is a simple scene for matching due to slow motion. In the helicopter, cars7 and cars8 sequences, Subset produces useless results. Table 4 shows that our method is significantly better than Subset in terms of segmentation accuracy on the helicopter scene. Although the baseline achieves the lowest error, it must be noted that it does not provide a useful solution to segmentation since it classifies less than $50\%$ of the points. This can also be seen in Fig. 21 where the baseline is not able to classify any point in the moving object in 5 out of 10 images.

5 Conclusion

We presented a new solution to the motion segmentation where the problem is split in two steps. First, a segmentation is performed independently on pairs of images. Then, the partial/local results are combined to segment points in all the images. This general framework – combined with a robust solution to two-frame motion segmentation (e.g. RPA [28]) – handles realistic situations such as the presence of mismatches that have been overlooked so far in previous motion segmentation work. Our approach does not require tracks as input but only pairwise correspondences. Thus it could be exploited to build tracks that are aware of segmentation, which constitutes the foundation of a multibody structure from motion pipeline. Future research will explore this direction.

Acknowledgements.

The authors would like to thank Luca Magri for his guidance through the Matlab code of RPA [28] and Stanislav Steidl for his help with the experiments. This work was supported by the European Regional Development Fund under the project IMPACT (reg. no CZ$.02.1.01/0.0/0.0/15\_003/0000468$).