Refractive Light-Field Features for Curved Transparent Objects in Structure from Motion

TL;DR

This paper introduces a novel light-field feature detection method for curved transparent objects, significantly improving structure-from-motion accuracy and convergence in challenging scenes involving refractive materials.

Contribution

The paper presents a new light-field feature that captures refracted patterns, enabling better 3D reconstruction and camera pose estimation for transparent, curved objects.

Findings

Improved camera pose estimates in refractive scenes

15-35% higher convergence rate than state-of-the-art

Enhanced 3D reconstructions of transparent objects

Abstract

Curved refractive objects are common in the human environment, and have a complex visual appearance that can cause robotic vision algorithms to fail. Light-field cameras allow us to address this challenge by capturing the view-dependent appearance of such objects in a single exposure. We propose a novel image feature for light fields that detects and describes the patterns of light refracted through curved transparent objects. We derive characteristic points based on these features allowing them to be used in place of conventional 2D features. Using our features, we demonstrate improved structure-from-motion performance in challenging scenes containing refractive objects, including quantitative evaluations that show improved camera pose estimates and 3D reconstructions. Additionally, our methods converge 15-35% more frequently than the state-of-the-art. Our method is a critical step towards allowing robots to operate around refractive objects, with applications in manufacturing, quality assurance, pick-and-place, and domestic robots working with acrylic, glass and other transparent materials.