Coherent Motion Segmentation in Moving Camera Videos using Optical Flow Orientations

TL;DR

This paper presents a novel optical flow orientation-based motion segmentation method for moving camera videos that accurately clusters pixels by real-world motion regardless of depth, outperforming traditional pixel-based approaches.

Contribution

It introduces a probabilistic model leveraging optical flow orientations to distinguish motions independent of depth, improving segmentation accuracy in complex scenes.

Findings

Successfully segments static objects at different depths as a single motion

Robust performance demonstrated on over thirty benchmark videos

Outperforms traditional pixel-based motion segmentation methods

Abstract

In moving camera videos, motion segmentation is commonly performed using the image plane motion of pixels, or optical flow. However, objects that are at different depths from the camera can exhibit different optical flows even if they share the same real-world motion. This can cause a depth-dependent segmentation of the scene. Our goal is to develop a segmentation algorithm that clusters pixels that have similar real-world motion irrespective of their depth in the scene. Our solution uses optical flow orientations instead of the complete vectors and exploits the well-known property that under camera translation, optical flow orientations are independent of object depth. We introduce a probabilistic model that automatically estimates the number of observed independent motions and results in a labeling that is consistent with real-world motion in the scene. The result of our system is that static objects are correctly identified as one segment, even if they are at different depths. Color features and information from previous frames in the video sequence are used to correct occasional errors due to the orientation-based segmentation. We present results on more than thirty videos from different benchmarks. The system is particularly robust on complex background scenes containing objects at significantly different depths