Trajectory Representation and Landmark Projection for Continuous-Time Structure from Motion

TL;DR

This paper enhances continuous-time structure from motion by introducing spline-based trajectory modeling, analyzing interpolation methods, and addressing landmark reprojection for rolling shutter cameras, leading to improved convergence and efficiency.

Contribution

It proposes a $ ext{C}^2$-continuous spline formulation that naturally incorporates inertial data and analyzes different interpolation strategies, providing insights into their effects on motion estimation.

Findings

Split interpolation on $ ext{SO}(3)$ and $ ext{R}^3$ outperforms $ ext{SE}(3)$ in tests.

Joint interpolation on $ ext{SE}(3)$ couples translation and rotation, which may not suit all scenarios.

Reprojection methods for rolling shutter cameras have similar quality but different computational costs.

Abstract

This paper revisits the problem of continuous-time structure from motion, and introduces a number of extensions that improve convergence and efficiency. The formulation with a $\mathcal{C}^2$-continuous spline for the trajectory naturally incorporates inertial measurements, as derivatives of the sought trajectory. We analyse the behaviour of split interpolation on $\mathbb{SO}(3)$ and on $\mathbb{R}^3$, and a joint interpolation on $\mathbb{SE}(3)$, and show that the latter implicitly couples the direction of translation and rotation. Such an assumption can make good sense for a camera mounted on a robot arm, but not for hand-held or body-mounted cameras. Our experiments show that split interpolation on $\mathbb{SO}(3)$ and on $\mathbb{R}^3$ is preferable over $\mathbb{SE}(3)$ interpolation in all tested cases. Finally, we investigate the problem of landmark reprojection on rolling shutter cameras, and show that the tested reprojection methods give similar quality, while their computational load varies by a factor of 2.