Semantic Texture for Robust Dense Tracking

TL;DR

This paper introduces a dense tracking method that uses CNN-derived semantic textures for more robust SLAM, outperforming traditional RGB-based approaches especially under lighting variations.

Contribution

It proposes replacing the standard image pyramid with CNN feature hierarchies for dense alignment, enhancing robustness and efficiency in visual SLAM.

Findings

Robust camera rotation tracking over time-lapse sequences.

CNN feature hierarchy improves alignment robustness to lighting changes.

Selective feature use maintains accuracy while increasing efficiency.

Abstract

We argue that robust dense SLAM systems can make valuable use of the layers of features coming from a standard CNN as a pyramid of `semantic texture' which is suitable for dense alignment while being much more robust to nuisance factors such as lighting than raw RGB values. We use a straightforward Lucas-Kanade formulation of image alignment, with a schedule of iterations over the coarse-to-fine levels of a pyramid, and simply replace the usual image pyramid by the hierarchy of convolutional feature maps from a pre-trained CNN. The resulting dense alignment performance is much more robust to lighting and other variations, as we show by camera rotation tracking experiments on time-lapse sequences captured over many hours. Looking towards the future of scene representation for real-time visual SLAM, we further demonstrate that a selection using simple criteria of a small number of the total set of features output by a CNN gives just as accurate but much more efficient tracking performance.