A Multi-cut Formulation for Joint Segmentation and Tracking of Multiple Objects

TL;DR

This paper introduces a joint multicut formulation that integrates point trajectory data and object detection cues to simultaneously improve motion segmentation and multi-target tracking.

Contribution

It proposes a novel joint graphical model that combines local trajectory relationships with high-level detection information for unified segmentation and tracking.

Findings

Improved results on FBMS59 motion segmentation benchmark.

Enhanced pedestrian tracking performance on 2D MOT 2015 sequences.

Demonstrates the benefit of combining local and high-level cues in a unified model.

Abstract

Recently, Minimum Cost Multicut Formulations have been proposed and proven to be successful in both motion trajectory segmentation and multi-target tracking scenarios. Both tasks benefit from decomposing a graphical model into an optimal number of connected components based on attractive and repulsive pairwise terms. The two tasks are formulated on different levels of granularity and, accordingly, leverage mostly local information for motion segmentation and mostly high-level information for multi-target tracking. In this paper we argue that point trajectories and their local relationships can contribute to the high-level task of multi-target tracking and also argue that high-level cues from object detection and tracking are helpful to solve motion segmentation. We propose a joint graphical model for point trajectories and object detections whose Multicuts are solutions to motion segmentation {\it and} multi-target tracking problems at once. Results on the FBMS59 motion segmentation benchmark as well as on pedestrian tracking sequences from the 2D MOT 2015 benchmark demonstrate the promise of this joint approach.