Pixel-aligned RGB-NIR Stereo Imaging and Dataset for Robot Vision

TL;DR

This paper presents a new robotic vision system with pixel-aligned RGB-NIR stereo cameras and a dataset capturing diverse lighting conditions, enabling improved 3D vision through novel fusion methods.

Contribution

It introduces a novel pixel-aligned RGB-NIR stereo imaging system, a synchronized dataset, and two fusion methods for enhanced robotic vision in challenging lighting.

Findings

Pixel-aligned RGB-NIR images improve vision tasks.

Fusion methods outperform single-spectrum approaches.

Dataset enables robust vision under diverse lighting.

Abstract

Integrating RGB and NIR stereo imaging provides complementary spectral information, potentially enhancing robotic 3D vision in challenging lighting conditions. However, existing datasets and imaging systems lack pixel-level alignment between RGB and NIR images, posing challenges for downstream vision tasks. In this paper, we introduce a robotic vision system equipped with pixel-aligned RGB-NIR stereo cameras and a LiDAR sensor mounted on a mobile robot. The system simultaneously captures pixel-aligned pairs of RGB stereo images, NIR stereo images, and temporally synchronized LiDAR points. Utilizing the mobility of the robot, we present a dataset containing continuous video frames under diverse lighting conditions. We then introduce two methods that utilize the pixel-aligned RGB-NIR images: an RGB-NIR image fusion method and a feature fusion method. The first approach enables existing RGB-pretrained vision models to directly utilize RGB-NIR information without fine-tuning. The second approach fine-tunes existing vision models to more effectively utilize RGB-NIR information. Experimental results demonstrate the effectiveness of using pixel-aligned RGB-NIR images across diverse lighting conditions.