Neural Camera Models

TL;DR

This paper explores neural camera models to improve depth sensing in robotics, addressing challenges like unreliable camera info and diverse camera types, aiming to make cameras more universally effective for depth estimation.

Contribution

It introduces methods to relax common assumptions in depth estimation, enhancing the versatility and reliability of neural camera models for real-world applications.

Findings

Improved depth estimation accuracy across diverse camera types.

Reduced reliance on ground truth depth labels.

Enhanced robustness to unreliable camera information.

Abstract

Modern computer vision has moved beyond the domain of internet photo collections and into the physical world, guiding camera-equipped robots and autonomous cars through unstructured environments. To enable these embodied agents to interact with real-world objects, cameras are increasingly being used as depth sensors, reconstructing the environment for a variety of downstream reasoning tasks. Machine-learning-aided depth perception, or depth estimation, predicts for each pixel in an image the distance to the imaged scene point. While impressive strides have been made in depth estimation, significant challenges remain: (1) ground truth depth labels are difficult and expensive to collect at scale, (2) camera information is typically assumed to be known, but is often unreliable and (3) restrictive camera assumptions are common, even though a great variety of camera types and lenses are used in practice. In this thesis, we focus on relaxing these assumptions, and describe contributions toward the ultimate goal of turning cameras into truly generic depth sensors.