Motion Segmentation from a Moving Monocular Camera

TL;DR

This paper presents a novel approach for motion segmentation from a moving monocular camera by combining point trajectory and optical flow cues, enabling accurate segmentation of complex motions in challenging scenes.

Contribution

It introduces a method that synergistically fuses point trajectory and optical flow information at the object level using co-regularized spectral clustering, achieving state-of-the-art results.

Findings

State-of-the-art performance on KT3DMoSeg dataset

Effective modeling of complex object motions

Improved segmentation accuracy in challenging scenes

Abstract

Identifying and segmenting moving objects from a moving monocular camera is difficult when there is unknown camera motion, different types of object motions and complex scene structures. To tackle these challenges, we take advantage of two popular branches of monocular motion segmentation approaches: point trajectory based and optical flow based methods, by synergistically fusing these two highly complementary motion cues at object level. By doing this, we are able to model various complex object motions in different scene structures at once, which has not been achieved by existing methods. We first obtain object-specific point trajectories and optical flow mask for each common object in the video, by leveraging the recent foundational models in object recognition, segmentation and tracking. We then construct two robust affinity matrices representing the pairwise object motion affinities throughout the whole video using epipolar geometry and the motion information provided by optical flow. Finally, co-regularized multi-view spectral clustering is used to fuse the two affinity matrices and obtain the final clustering. Our method shows state-of-the-art performance on the KT3DMoSeg dataset, which contains complex motions and scene structures. Being able to identify moving objects allows us to remove them for map building when using visual SLAM or SFM.