3D Trajectory Reconstruction of Dynamic Objects Using Planarity Constraints

TL;DR

This paper introduces a novel method for 3D trajectory reconstruction of dynamic objects in monocular videos, leveraging shape tracking, structure from motion, and planarity constraints, validated on a new urban drone dataset.

Contribution

It combines semantic segmentation, optical flow, and environment modeling to accurately reconstruct object trajectories, addressing occlusion and stationary objects.

Findings

Achieves an average reconstruction error of 0.31 meters.

Robust to occlusion and stationary objects.

Introduces a new urban drone dataset for evaluation.

Abstract

We present a method to reconstruct the three-dimensional trajectory of a moving instance of a known object category in monocular video data. We track the two-dimensional shape of objects on pixel level exploiting instance-aware semantic segmentation techniques and optical flow cues. We apply Structure from Motion techniques to object and background images to determine for each frame camera poses relative to object instances and background structures. By combining object and background camera pose information, we restrict the object trajectory to a one-parameter family of possible solutions. We compute a ground representation by fusing background structures and corresponding semantic segmentations. This allows us to determine an object trajectory consistent to image observations and reconstructed environment model. Our method is robust to occlusion and handles temporarily stationary objects. We show qualitative results using drone imagery. Due to the lack of suitable benchmark datasets we present a new dataset to evaluate the quality of reconstructed three-dimensional object trajectories. The video sequences contain vehicles in urban areas and are rendered using the path-tracing render engine Cycles to achieve realistic results. We perform a quantitative evaluation of the presented approach using this dataset. Our algorithm achieves an average reconstruction-to-ground-truth distance of 0.31 meter.