Nonrigid Optical Flow Ground Truth for Real-World Scenes with Time-Varying Shading Effects

TL;DR

This paper introduces a new real-world dataset with ground truth for nonrigid scene deformation, enabling better evaluation of optical flow algorithms under realistic conditions including lighting and occlusion effects.

Contribution

It provides the first real-world deforming scene dataset with ground truth using NIR markers, and proposes a multispectral optical flow model combining RGB and NIR data.

Findings

Eight RGB optical flow methods evaluated on the new dataset.

Hybrid RGB-NIR optical flow outperforms purely RGB-based methods.

The dataset captures realistic photometric effects like blur and illumination change.

Abstract

In this paper we present a dense ground truth dataset of nonrigidly deforming real-world scenes. Our dataset contains both long and short video sequences, and enables the quantitatively evaluation for RGB based tracking and registration methods. To construct ground truth for the RGB sequences, we simultaneously capture Near-Infrared (NIR) image sequences where dense markers - visible only in NIR - represent ground truth positions. This allows for comparison with automatically tracked RGB positions and the formation of error metrics. Most previous datasets containing nonrigidly deforming sequences are based on synthetic data. Our capture protocol enables us to acquire real-world deforming objects with realistic photometric effects - such as blur and illumination change - as well as occlusion and complex deformations. A public evaluation website is constructed to allow for ranking of RGB image based optical flow and other dense tracking algorithms, with various statistical measures. Furthermore, we present an RGB-NIR multispectral optical flow model allowing for energy optimization by adoptively combining featured information from both the RGB and the complementary NIR channels. In our experiments we evaluate eight existing RGB based optical flow methods on our new dataset. We also evaluate our hybrid optical flow algorithm by comparing to two existing multispectral approaches, as well as varying our input channels across RGB, NIR and RGB-NIR.