TL;DR
This paper introduces a learning-based method for segmenting moving objects in videos by combining motion cues from optical flow with appearance features, outperforming prior approaches on multiple benchmarks.
Contribution
It presents a novel spatio-temporal grouping approach that leverages motion and appearance cues, and introduces new benchmarks for generic moving object detection.
Findings
Outperforms all prior work on FBMS dataset
Matches top-down methods on common categories
Significantly outperforms on unseen categories
Abstract
Detecting and segmenting individual objects, regardless of their category, is crucial for many applications such as action detection or robotic interaction. While this problem has been well-studied under the classic formulation of spatio-temporal grouping, state-of-the-art approaches do not make use of learning-based methods. To bridge this gap, we propose a simple learning-based approach for spatio-temporal grouping. Our approach leverages motion cues from optical flow as a bottom-up signal for separating objects from each other. Motion cues are then combined with appearance cues that provide a generic objectness prior for capturing the full extent of objects. We show that our approach outperforms all prior work on the benchmark FBMS dataset. One potential worry with learning-based methods is that they might overfit to the particular type of objects that they have been trained on. To…
Click any figure to enlarge with its caption.
Figure 1
Figure 2
Figure 1
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8
Figure 9| Flow type | Det @ 0.5 | Seg @ 0.5 |
|---|---|---|
| FlowNet2 | 40.5 | 23.9 |
| FlowNet2 Groundtruth | 43.2 | 24.1 |
| LiteFlowNet Groundtruth | 33.8 | 24.0 |
| COCO Training | Det @ 0.5 | Seg @ 0.5 |
|---|---|---|
| Class-specific | 42.0 | 40.2 |
| Objectness | 49.8 | 48.3 |
| Variant | Det @ 0.5 | Seg @ 0.5 |
|---|---|---|
| Joint Training, class-specific | 73.8 | 70.3 |
| Joint Training, objectness | 79.1 | 73.3 |
| + Freeze appearance | 81.9 | 76.7 |
| + Freeze motion | 83.7 | 76.4 |
| + Freeze mask | 83.9 | 77.4 |
| Joint Training Data | Det @ 0.5 | Seg @ 0.5 |
|---|---|---|
| DAVIS | 83.9 | 77.4 |
| YTVOS-moving | 79.9 | 75.8 |
| DAVIS YTVOS-moving | 85.1 | 77.9 |
| P | R | F | |
|---|---|---|---|
| [20] | 39.4 | 53.8 | 42.3 |
| Mask R-CNN | 70.8 | 75.6 | 71.6 |
| Ours-A | 67.7 | 77.1 | 70.9 |
| Ours | 78.3 | 78.8 | 78.1 |
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Towards Segmenting Anything That Moves
Achal Dave
Carnegie Mellon University
Pavel Tokmakov
Carnegie Mellon University
Deva Ramanan
Carnegie Mellon University
Abstract
Detecting and segmenting individual objects, regardless of their category, is crucial for many applications such as action detection or robotic interaction. While this problem has been well-studied under the classic formulation of spatio-temporal grouping, state-of-the-art approaches do not make use of learning-based methods. To bridge this gap, we propose a simple learning-based approach for spatio-temporal grouping. Our approach leverages motion cues from optical flow as a bottom-up signal for separating objects from each other. Motion cues are then combined with appearance cues that provide a generic objectness prior for capturing the full extent of objects. We show that our approach outperforms all prior work on the benchmark FBMS dataset. One potential worry with learning-based methods is that they might overfit to the particular type of objects that they have been trained on. To address this concern, we propose two new benchmarks for generic, moving object detection, and show that our model matches top-down methods on common categories, while out-performing both top-down and bottom-up methods on never-before-seen categories.
1 Introduction
People have the remarkable ability to thrive in a staggeringly diverse world, frequently encountering things they have never seen before. Our approaches for machine perception, meanwhile, often remain trapped in a closed world, as in the case of object recognition, where approaches are designed to recognize and name one of pre-defined classes. But practical robot autonomy requires robust perception in the open-world: even a self-driving car must be able to detect never-before-seen obstacles and debris, regardless of what particular semantic name it happens to associate with.
In the computer vision community, open-world recognition is typically addressed from a machine-learning perspective such as zero-shot learning [37] or open-set classification [34]. We advocate a different approach that has its roots in classic vision: perceptual grouping. Specifically, we wish to segment out all moving object instances in a video stream, including never-before-seen object categories. Defining the notion of a generic, never-before-seen object is notoriously challenging [1]. We intentionally focus on moving objects so as to take advantage of the “common fate” principle of grouping: pixels that move together should tend to be grouped together into objects [28].
Indeed, the problem of spatio-temporal grouping is a classic “mid-level” visual understanding task, dating back to the iconic work of Marr [24, 44]. Pre-deep learning solutions tend to follow bottom-up computational strategies for self-organization and clustering, often of long-term pixel trajectories [27, 20]. In the static image case, pixels can grouped by relying on Gestaltian notions of appearance similarity and curvilinear edge continuity [28]. One long-standing challenge in perceptual organization has been operationalizing these cues into an accurate algorithm for spatio-temporal grouping. Our key observation is that many of the recent advances in closed-world instance segmentation can be repurposed for open-world spatio-temporal grouping.
We first validate the performance of our proposed approach on the Freiburg Berkeley Motion Segmentation benchmark (FBMS). Because the standard measure used in FBMS does not penalize false positives, we find that trivial solutions can score well. We analyze the official metric in detail and propose a new, more informative evaluation. We achieve state-of-the-art results on both measures, and specifically outperform the next-best method of Keuper et al. [20] by 11.4% on our proposed measure.
To further study our method, we introduce the DAVIS-Moving and YTVOS-Moving benchmarks for motion-based grouping. We create these by selecting videos from the DAVIS 2017 [32] and YTVOS [46] datasets where all moving objects are labeled. On these new benchmarks, we strongly outperform top-down, closed world methods such as Mask R-CNN, as well as traditional bottom-up grouping methods. In particular, our approach is competitive with a top-down method for categories seen during training, but outperforms both top-down and bottom-up approaches for unseen categories.
To sum up, our contributions are three-fold: (1) we propose the first deep learning-based method for spatio-temporal grouping; (2) we propose a more informative metric and larger, more diverse benchmarks to enable further progress; (3) we report state-of-the-art results on the FBMS dataset and our larger, proposed benchmarks. The code and trained models will be made publicly available.
2 Related Work
Spatio-temporal grouping: Segmenting and tracking objects based on their motion has a rich history. An early work [35] proposed treating this task as a spatio-temporal grouping problem, a philosophy espoused by a number of more recent approaches, including [11, 6, 20], as well as [27], which introduced FBMS. In particular, these methods track each pixel individually with optical flow, encode the motion information of a pixel in a compact descriptor and then obtain an instance segmentation by clustering the pixels based on motion similarity. Unlike these works, our approach is driven primarily by a top-down learning algorithm followed by a simple linking step to generate spatio-temporal segmentations. The most relevant approach in this respect is [9], which trains a CNN to detect (but not segment) moving objects, and combines these detections with clustered pixel trajectories to derive segmentations. By contrast, our approach directly outputs segmentations at each frame, which we link together with an efficient tracker. Very recently, Bideau et al. [5] proposed to combine a heuristic-based motion segmentation method [26, 4] with a CNN trained for semantic segmentation for the task of moving object segmentation. Their method, however, does not handle discontinuous motion. In addition, the fact that they rely strongly on heuristic motion estimates allows our learning-based approach to outperform their method on FBMS by a wide margin. In very recent work, Xie et al. [45] introduced a deep learning approach for motion segmentation that segments and tracks moving objects using a recurrent neural network. By comparison, our method uses a simple, overlap-based tracker that performs competitively with the learned tracker from [45] while producing significantly fewer false positive segmentations (see Supplementary).
Foreground/Background Video Segmentation: Several works have focused on the binary version of the video segmentation task, separating all the moving objects from the background. Early approaches [8, 29, 42, 22] relied on heuristics in the optical flow field, such as closed motion boundaries in [29] to identified moving objects. These initial estimates were then refined with appearance, utilizing external cues, such as saliency maps [42], or object shape estimates [22]. Another line of work focused on building probabilistic models of moving objects using optical flow orientations [26, 4]. None of these methods are based on a robust learning framework and struggle to generalize well to unseen videos. The recent introduction of a standard benchmark, DAVIS 2016 [31], has led to a renewed interest. More recent approaches propose deep models for directly estimating motion masks, as in [18, 39, 40]. These approaches are similar to ours in that they also use a two-stream architecture to separately process motion and appearance, but they are unable to segment individual object instances, one of our primary goals. Our method separately segments and tracks each individual moving object in a video.
Object Detection: The task of segmenting object instances from still images has seen immense success in recent years, bolstered by large, standard datasets such as COCO [23]. However, this standard task focuses on segmenting every instance of objects belonging to a fixed list of categories, leading to methods that are designed to be blind to objects that fall outside the categories in the training set.
Two recent works have focused on extending these models to detect generic objects. [15] aims to generalize segmentation models to new categories, but requires bounding box annotations for each new category. More relevant to our approach, [19] aims to detect all “object”-like regions in an image, outputting a binary objectness mask. While we share their goal of segmenting unseen objects, our approach additionally provides instance masks for each object.
3 Approach
We propose a two-stream spatio-temporal grouping method that uses appearance and motion cues to segment all moving objects in a video. Our approach, illustrated in Figure 3, takes a frame together with a corresponding optical flow as input, and passes them through an “appearance stream” (top) and a “motion stream” (bottom) respectively. The resulting features are combined and passed to the joint region proposal network (RPN), which learn to detect and segment moving objects irrespective of their category.
Our approach shares inspiration with prior work that proposes two-stream approaches for object detection [12, 30, 10, 9], with two key differences. First, we design a novel region proposal module that learns to fuse both appearance and motion information to generate moving object detections. Second, to overcome the dearth of appropriate training data, we develop a stage-wise training strategy that allows us to leverage synthetic data to train our motion stream, image datasets to train our appearance stream, and a small amount of real video data to train the joint model.
We first discuss the architecture and training strategy for the motion and appearance streams individually, and then detail how to combine these streams into one coherent architecture. Finally, we describe a simple tracker that we use for linking detections across time, allowing us to produce spatio-temporal groupings that span across many frames.
3.1 Motion-based Segmentation
We start by training a motion-based instance segmentation model. As mentioned above, this requires videos with segmentation masks for all moving objects, which is difficult to obtain. Fortunately, prior work has shown that synthetic data can be used for some low-level tasks, such as flow estimation [7] and binary motion segmentation [39]. Inspired by this, we train our motion stream on the FlyingThings3D dataset [25], which contains nearly 2,700 synthetically generated sequences of 3D objects traveling in randomized trajectories, captured with a camera also traveling along a random trajectory. The dataset provides groundtruth optical flow, as well as segmentations for both static and moving objects (See Figure 2). We train our motion-stream using the moving instance labels from [39], treating all moving objects as a single category, and all other pixels, including static objects, as background. The resulting model learns to segment moving objects irrespective of their category. In fact, this model is oblivious to the whole notion of an object and is capable of segmenting parts that exhibit independent motion (see Figure 5). We discuss more details and variants of this approach in Section 5.3.1.
3.2 Appearance-based Segmentation
In order to incorporate appearance information, we next train an image-based object segmentation model that aims to segment the full extent of generic objects. Fortunately, large datasets exist for training image-based instance segmentation models. Here, we train on the MS COCO dataset [23], which contains approximately 120,000 training images with instance segmentation masks for each object in 80 categories. We could train our appearance stream following the standard Mask R-CNN training procedure, which jointly localizes and classifies each object in an image belonging to the 80 categories. However, this results in a model that, while proficient at segmenting 80 categories, is blind to objects from any other, novel category. Instead, we train an “objectness” Mask R-CNN by combining each of the 80 categories into a single “object” category. In Section 5.3.2, we will show that this “objectness” training (1) provides a significant improvement over standard training, and (2) leads to a model that generalizes surprisingly well to objects that are not labeled in MS COCO.
3.3 Two-Stream Model
Equipped with the individual appearance and motion streams, we now propose a two-stream architecture for fusing these information sources. In order to clearly describe our two-stream model, we take a brief detour to describe the Mask R-CNN architecture. Mask R-CNN contains three stages: (1) Feature extraction: a “backbone” network, such as ResNet [14], is used to extract features from an image. (2) Region proposal: A region proposal layer uses these features to selects regions likely to contain an object. Finally, (3) Regression: for each proposed region, the corresponding backbone features are pooled to a fixed size, and fed as input to bounding box and mask regression heads.
To build a two-stream instance segmentation model, we extract the backbone from our individual appearance-based and motion-based segmentation models. Next, as depicted in Figure 3, we propose a “two-stream” RPN that uses these two backbones, instead of a single backbone, to predict proposals from spatio-temporal features, extracted from the optical flow (blue) and RGB (orange) backbones. These features are concatenated and fed to a short series of convolutional layers to reduce the dimensionality to match that of Mask R-CNN, allowing us to maintain the architecture of stages (2) and (3). Intuitively, we expect the appearance stream to behave as a generic object detector, and our motion stream to help detect novel objects that the appearance stream may miss and filter out static objects.
Although this may appear similar to prior approaches for building a two-stream detection model, it differs in a key detail: prior approaches obtain region proposals either only from appearance features [12, 10, 9], or from appearance and motion features individually [30]. By contrast, we propose a novel proposal module that learns to fuse motion and appearance features to find object-like regions.
We train our joint model on subsets of the DAVIS and YouTube Video Object Segmentation datasets (as detailed in Section 5.1). We experiment with various strategies for training this joint model in Section 5.3.3.
3.4 Tracking
So far, we have focused on segmenting moving objects in each frame of a video. To maintain object identities and to continue segmenting objects after they stop moving, we implement a simple, overlap-based tracker inspired by [2]. First, we remove all detections with score below . On the first frame, all high scoring detections (score ) are used to initialize a track, which we define simply as a sequence of linked detections. At each successive frame, we compute the mask intersection over union between the most recent segmentation for each active track and predicted objects at , and use Hungarian Matching to assign predicted objects to tracks. Unmatched predictions are discarded if their score is ; else, they are used to initialize a new track. Tracks that have not been assigned a new object for up to frames are marked as inactive.
Tracking static objects: To continue tracking moving objects when they stop moving, we need to be able to detect static objects. A naïve way to do this is to run the objectness model trained in Section 3.2 in parallel with our two-stream model at every frame. However, this would be computationally expensive. Fortunately, our appearance stream shares the backbone of the objectness model. Thus, we only need to apply the (inexpensive) stages (2) and (3) of the objectness model on the appearance features extracted by our two-stream network. Using this, we can efficiently output a set of moving and static object predictions for each frame in a video. We merge the two outputs by removing any predicted static object that overlaps with a predicted moving object. We use the same tracker described above, using only moving objects to initialize tracks.
4 Evaluation
To evaluate methods for spatio-temporal grouping, we desire a metric that rewards segmenting and tracking moving objects, but penalizes the detection of static objects or background. While there has been a rich line of prior work related to our goal, standard metrics surprisingly do not satisfy these criterion. We propose a novel metric that does.
The default metric in FBMS [27] was designed for grouping-based approaches, but does not penalize false positive predictions. Recently, Bideau et al. [3] tackled this issue by measuring the difference between the number of groundtruth moving objects and the number of predicted moving objects ( Obj). However, this complicates method comparisons by relying on two separate metrics; instead, we propose a single and intuitive F-measure that evaluates a method’s ability to detect all and only moving objects.
Figure 4 (middle) visualizes the default FBMS metric which matches each predicted segment with a groundtruth segment so as to maximize IoU overlap, ignoring any unmatched predictions. This means the default F-measure does not penalize false positive segments, unfairly favoring methods that generate a large number of predictions. By contrast, our proposed F-measure, depicted in Figure 4 (right), counts unmatched predictions as false positives.
More precisely, we describe our metric roughly following the notation in [27]. For each video, let be the pixels belonging to a predicted region , and be all the pixels belonging to a groundtruth non-background region . While [27] omits unlabeled pixels from evaluation, we include all pixels in the groundtruth.
Let be the precision, be the recall, and be the F-measure corresponding to this pair of predicted and groundtruth regions, as follows:
[TABLE]
Following [27], we use the Hungarian algorithm to find a matching between predictions and groundtruth that maximizes the sum of the F-measure over all assignments. Let be the groundtruth matched to each predicted region; for any that is not matched to a groundtruth cluster, is set to an empty region. We define our metric as follows:
[TABLE]
Any unlabeled pixel in a predicted region will reduce precision and F-measure, penalizing the segmentation of static or unlabeled objects. In our experiments, we report results with both the official and our proposed measure.
5 Experiments
We first analyze each component of our proposed model with experimental results. Next, we compare our approach to prior work in spatio-temporal grouping on three datasets.
5.1 Datasets
An ideal dataset for training our model would contain a large number of videos where every moving object has labeled instance masks, and static objects are not labeled. Three candidate datasets exist for this task: YouTube Video Object Segmentation (YTVOS) [46], DAVIS 2016 [31], and FBMS [27]. While YTVOS contains over 3,000 short videos with instance segmentation labels, not all objects in these videos are necessarily labeled, and both moving as well as static objects may be labeled. The DAVIS 2016 dataset contains instance segmentation masks (provided with DAVIS 2017) for only the moving objects, but only contains 30 training videos. Finally, although FBMS contains a total of 59 sequences with labeled instance segmentation masks for moving objects, prior work evaluates on the entire dataset, preventing us from training on any sequences in the dataset in order to provide a fair comparison.
To overcome this lack of data, we use heterogeneous data sources to train our model in a stagewise fashion. As described earlier, we train our appearance stream on COCO [23]. We train our motion stream on FlyingThings3D [25], a synthetic dataset of 2,700 videos of randomly moving 3D objects. Finally, we fine-tune our joint model on DAVIS2016 and the training subset of YTVOS-Moving. We use a held-out set of 100 YTVOS-Moving sequences for evaluation.
5.2 Implementation Details
Network Architecture: Our two-stream model is built off Mask R-CNN [13] with a ResNet-50 backbone. We will publicly release the code and exact configuration for training, highlight some important details here, and note further details in supplementary. All our models are trained using the publicly available PyTorch implementation of Detectron [41]. In general, we use the original hyper-parameters provided by the authors of Mask R-CNN. The backbone for every model is pre-trained on ImageNet [33]. When constructing our two-stream model, we initialize the bounding box and mask heads from the appearance-only model.
Tracking: We set the confidence threshold for initializing tracks, as described in Section 3.4, to , and remove any detections with confidence lower than . We allow tracks to stay alive for up to frames (approximately for most videos), although we found the final results are fairly insensitive to this parameter. To detect objects before they move, we first run our tracker forwards, and then backwards in time.
5.3 Ablation analysis
Evaluation: We analyze our model by benchmarking various configurations on the DAVIS 2016 dataset [31]. For ablation, we found it helpful to use the standard detection mean average precision (mAP) metric [23] in place of video object segmentation metrics, which require tracking and obfuscate analysis of our architecture choices. We report both detection and segmentation mAP at an IoU threshold of 0.5
5.3.1 Motion stream
To begin, we explore training strategies for the motion stream of our model. We train our motion stream on the FlyingThings3D dataset, as described in Section 3.1. This dataset provides groundtruth flow, which we could use for training. However, at inference time, we only have access to noisy, estimated flow. In order to match flow in the real world, we estimate flow on FlyingThings3D using two optical flow estimation methods: FlowNet2 and LiteFlowNet. For both methods, we use the version of their model that is trained on synthetic data and fine-tuned on real data.
In Table 1, we compare three strategies for training on FlyingThings3D. We start by training using only FlowNet2 flow as input (“FlowNet2”). We hypothesize that training directly on noisy, estimated flow can lead to difficulties in early training. To overcome this, we train a variant starting with groundtruth flow, and fine-tune on FlowNet2 flow (“FlowNet2 Groundtruth” row). We find that this provides a significant improvement (). We also considered using a more recent flow estimation method, LiteFlowNet [16] (“LiteFlowNet Groundtruth” row). Surprisingly, we find that FlowNet2 provides significant improvements for detection, despite performing worse on standard flow estimation benchmarks. Qualitatively, we found that FlowNet2 provides sharper results along boundaries than LiteFlowNet, which may aid in localizing objects.
Figure 5 shows qualitative results of the motion stream. Despite never having seen real images with segmentation labels, this model is able to group together parts that move alike, while separating objects with disparate motion.
5.3.2 Appearance Stream
While our motion stream is proficient at grouping similarly-moving pixels, it lacks any priors for real world objects and will not hesitate to oversegment common objects, such as the man in Figure 5. To introduce these useful priors, we turn our attention to the appearance stream of our model.
As described in Section 3.2, we train our appearance stream on the COCO dataset [23]. We evaluate two variants of training. First, we train a standard, “class-specific” Mask R-CNN, that outputs a set of boxes and masks for each of the 80 categories in the COCO Dataset. At inference time, we combine the boxes and masks predicted for each category into a single “object” category. Second, we train an “objectness” Mask R-CNN, by collapsing all the categories in COCO to a single category before training.
We show results from these two variants in Table 2. Our “objectness” model significantly outperforms the standard “class-specific” model by nearly 8%. We further compare the two models qualitatively in Figure 6, noting that our objectness model better generalizes to non-COCO categories.
5.3.3 Joint training
Finally, we combine our appearance and flow streams in a single two-stream model, depicted in Figure 3 and described in detail in Section 3.3. We experiment with different strategies for training this joint model. Throughout these experiments, we initialize the flow stream with the “FlowNet2 Groundtruth” model from Section 5.3.1, and use the objectness model from Section 5.3.2 to initialize the appearance stream, the box and mask prediction heads, and the RPN. We show the results in Table 3.
We start by training this joint model directly on the DAVIS 2016 training set, which achieves 79.1% mAP. We note that even with joint-training, using the objectness model for initialization provides a significant boost over using a category-specific detector (73.8%). Next, to maintain the generalizability of the objectness model, we also train a variant where we freeze the weights of the appearance stream. This provides nearly a 3% improvement in accuracy. Similarly, to maintain the generic “grouping” nature of the synthetically-trained flow stream, we freeze the flow stream, providing us with an additional 2% improvement.
Finally, we hypothesize that while features from the flow stream are helpful for localizing generic moving objects, appearance information is sufficient for segmentation. We verify this hypothesis by training one last variant where the mask head uses only appearance stream features, and freeze its weights to those of the objectness model. Indeed, this provides a modest improvement of 1% in segmentation AP.
Training Data: Next, we train our joint model on YTVOS-Moving (Section 5.1) and show results in Table 4. Unfortunately, this dataset contains very few static objects, causing the model to detect both static and moving objects, leading to a significant (5%) drop in performance. However, fine-tuning this model on the DAVIS 16 training set leads to our best model (DAVIS YTVOS-moving).
5.4 Comparison to prior work
Official FBMS: We first evaluate our method against prior work on the standard FBMS benchmark in Table 5. As discussed in Section 5.1, this metric does not penalize false positive detections. As expected, our appearance stream alone, despite segmenting both static and moving objects, performs best on this metric (‘Ours-A’), outperforming all prior work by 6.4% in F-measure on the TestSet, and 2.2% on the TrainingSet 111Note that despite the name, we do not use either set for training.. For completion, we also report the performance of our joint model (‘Ours-J’), which compares favorably to state-of-the-art despite the flawed metric. Our improvements on this metric are likely driven by improvements in segmentation boundaries (see Figure 7).
Proposed FBMS: Finally, we report results on our proposed metric in Table 6. Recall that our proposed metric generally follows the official metric, but additionally penalizes detection of static objects. We compare to all methods from Table 5 whose final results on FBMS were accessible or provided by the authors through personal communication. On this proposed metric, we first note that, as expected, the performance of our appearance model baseline is significantly worse than our final, joint model, by 9.2% on TestSet and 6% on TrainingSet in F-measure. More importantly, our final model strongly out performs prior work in F-measure by 11.3% on the TestSet, and 6.1% on the TrainingSet. In addition to improving segmentation boundaries, our approach effectively removes spurious segmentations of background regions and object parts (Figure 7).
Qualitative results: We qualitatively compare our approach with Keuper et al. [20] and Bideau et al. [5] in Figure 7222 [5] only segments objects while they move. We provide an evaluation using an alternative FBMS labeling they propose in our supplementary. In the top row of Figure 7, [20] oversegments the dog into multiple parts, and [5] merges the dog with the background, whereas our approach fully segments the dog. Similarly, the cat in the middle row is over-segmented by [20] and under-segmented by [5], but well-segmented by our approach. In the final row, both [20] and [5] exhibit segmentation and tracking errors; the region corresponding to the man’s foot (colored yellow for Keuper et al. and red for Bideau et al.) are mistakenly tracked into a background region thus segmenting part of the background as a moving object. Meanwhile, our object-based tracker fully segments the person and the tennis racket with high precision. We show further qualitative results in supplementary material.
DAVIS-Moving: We further evaluate our method on a subset of the DAVIS 17 dataset. Unlike DAVIS 2016, the 2017 version provides instance-level masks for objects, but contains sequences with labeled static or unlabeled moving objects. For evaluation, we manually select 22 of 30 validation videos without these issues, and refer to this subset as DAVIS-Moving. We compare to [20], the best FBMS method we can obtain code for, with our proposed metric in Table 7. Surprisingly, we find a much larger gap in performance on this dataset; while [20] achieves 42.3% on F-measure with our proposed metric, our approach improves significantly to 77.9%. We believe this gap may be due to faster, more articulated motion and higher resolution videos in DAVIS 17, which severely affect [20] but not our method.
YTVOS-Moving: Finally, we evaluate on sequences from YTVOS-Moving (selected from YTVOS, as described in Section 5.1). Unlike FBMS and DAVIS, YTVOS contains diverse objects, such as octopuses and snakes. For fairness, we evaluate a version of our final model that was never trained on YTVOS, and show results in Table 8. In the first column, we report the F-measure overall on the YTVOS-Moving dataset. In the second column, we report F-measure on a subset of videos containing objects not from COCO categories, which our model has not seen during training. Overall, we find that Mask R-CNN struggles to detect the diverse objects in this data. We first show that our appearance stream alone (Ours-A) significantly improves over this baseline, from 53.6% to 69.0%. Our final approach (Ours-J) performs comparably overall to the appearance stream alone, suggesting that it is a key driver of our improvements on this data. On novel objects, however, our final approach provides modest improvements by incorporating the motion stream. We show further qualitative results in the appendix.
6 Conclusion
We proposed a simple learning-based approach for spatio-temporal grouping. Our method provides two key insights. First, learning based approaches are able to generalize to never-before-seen objects (Section 5.3.2). Second, synthetic data can be used to train a truly generic grouping method with little priors on real world objects. As a result, our approach achieves state-of-the-art results on the FBMS benchmark dataset. Finally, to enable further research in this direction, we introduced a new metric as well as two new benchmarks (DAVIS-Moving, YTVOS-Moving).
Acknowledgements: We thank Pia Bideau for providing evaluation code, Nadine Chang, Kenneth Marino and Senthil Purushwalkam for reviewing drafts and discussions. Supported by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior/Interior Business Center (DOI/IBC) contract number D17PC00345. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes not withstanding any copyright annotation theron. Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied of IARPA, DOI/IBC or the U.S. Government.
7 Appendix
We include an additional comparison to prior methods on DAVIS’16 motion segmentation in Section 7.1, to a prior method on FBMS in Section 7.2, and further implementation details in Section 7.4.
7.1 DAVIS’16 Motion Segmentation
We compare our method with motion foreground-background segmentation methods on DAVIS 2016. We convert our instance segmentation output into binary motion masks by marking as foreground any pixel belonging to a predicted instance with a score , and report results in Table 9. Although this is not our target task, our method compares favorably to the state of the art, and even modestly improves the F-measure for boundary accuracy () by 1%. On the metric it is only outperformed by a concurrent work by 0.7%.
7.2 FBMS Moving Only
A recent line of work [3, 5] proposed evaluating a subtask of spatiotemporal grouping. Whereas standard spatiotemporal grouping requires segmenting and tracking all instances that move at any point in the video, [5] focuses on segmenting and tracking instances only in frames where they move.
In order to evaluate this subtask, [5] uses an alternative labeling for FBMS introduced in [3], and supplements the official FBMS measure with a Obj metric, which indicates the average absolute difference between the number of predicted objects and groundtruth objects in each sequence. Intuitively, this penalizes false-positive detection of static objects or background; we refer the reader to [5] for further information. As with the official FBMS metric, the precision, recall and F-measure do not penalize static detections.
As described in Sec. 3.4 of our main submission, our final approach uses the appearance stream to track objects even after they stop moving. For a fair comparison, we disable this component, applying tracking directly to the output of our two-stream model that detects only moving objects in each frame.
Table 10 shows that our method significantly reduces the number of false positive segmentations compared to both [45, 5] as evidenced by the improvement in Obj of nearly 65% (from 4 to 1.4), while performing competitively with [45] on F-measure.
7.3 Failure cases
We also present illustrative failure cases of our method in Figure 8. The most common mistakes our method makes are tracking failures and the misclassification of static objects as moving. In the top row, the furthest horse (colored purple in the first frame) is completely occluded by the man in the middle frame, leading to an identity swap in the last frame, indicated by the color swap from purple to yellow. In the bottom row, the car further to the right is parked and not moving but is classified as moving by our approach. Our hypothesis, based on viewing similar failure cases, is that this is due to the proximity of the static car to the moving van, which our learning-based approach may have learnt to use as a cue for identifying moving objects.
7.4 Implementation details
Miscellaneous: We encode optical flow following [40]: We use a 3-channel image for ease of use with image-based CNNs, where the first channel encodes the angle and the second channel encodes the magnitude at each pixel, and the last channel is empty. For extracting flow, we use the version of FlowNet2 trained on synthetic data and fine-tuned on real data. For all visualizations throughout this paper, we use a confidence threshold of .
Training Regime: We train the appearance stream on COCO [23] to convergence (90,000 iterations). We train our motion stream on FlyingThings3D using Groundtruth flow as input, and then with FlowNet2 flow [17] for 10,000 iterations each. We train the joint model on YTVOS and then on DAVIS for 5,000 iterations each.
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1[1] Bogdan Alexe, Thomas Deselaers, and Vittorio Ferrari. What is an object? In CVPR , 2010.
- 2[2] Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft. Simple online and realtime tracking. In ICIP , 2016.
- 3[3] Pia Bideau and Erik Learned-Miller. A detailed rubric for motion segmentation. ar Xiv preprint ar Xiv:1610.10033 , 2016.
- 4[4] Pia Bideau and Erik Learned-Miller. It’s moving! a probabilistic model for causal motion segmentation in moving camera videos. In ECCV , 2016.
- 5[5] Pia Bideau, Aruni Roy Chowdhury, Rakesh R Menon, and Erik Learned-Miller. The best of both worlds: Combining CN Ns and geometric constraints for hierarchical motion segmentation. In CVPR , 2018.
- 6[6] Thomas Brox and Jitendra Malik. Object segmentation by long term analysis of point trajectories. In ECCV , 2010.
- 7[7] Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In ICCV , 2015.
- 8[8] Alon Faktor and Michal Irani. Video segmentation by non-local consensus voting. In BMVC , 2014.
