Unsupervised Learning of Depth Estimation and Visual Odometry for Sparse Light Field Cameras

TL;DR

This paper presents an unsupervised learning approach enabling robots to interpret and calibrate sparse light field cameras for improved depth estimation and odometry, facilitating integration of new imaging devices in robotics.

Contribution

It introduces a generalized encoding for sparse light fields allowing unsupervised learning of depth and odometry, outperforming traditional methods on 4D light field data.

Findings

Outperforms monocular and conventional techniques in depth and odometry accuracy

Provides metric scale depth and odometry estimates

Enables unsupervised recalibration over camera lifetime

Abstract

While an exciting diversity of new imaging devices is emerging that could dramatically improve robotic perception, the challenges of calibrating and interpreting these cameras have limited their uptake in the robotics community. In this work we generalise techniques from unsupervised learning to allow a robot to autonomously interpret new kinds of cameras. We consider emerging sparse light field (LF) cameras, which capture a subset of the 4D LF function describing the set of light rays passing through a plane. We introduce a generalised encoding of sparse LFs that allows unsupervised learning of odometry and depth. We demonstrate the proposed approach outperforming monocular and conventional techniques for dealing with 4D imagery, yielding more accurate odometry and depth maps and delivering these with metric scale. We anticipate our technique to generalise to a broad class of LF and sparse LF cameras, and to enable unsupervised recalibration for coping with shifts in camera behaviour over the lifetime of a robot. This work represents a first step toward streamlining the integration of new kinds of imaging devices in robotics applications.