Differential Scene Flow from Light Field Gradients

TL;DR

This paper introduces new methods for recovering detailed 3D scene flow from 4D light field data using differential analysis, enabling accurate motion sensing in various scenarios.

Contribution

It develops the first differential light field-based scene flow algorithms, including local, global, and combined methods, inspired by optical flow techniques.

Findings

High precision 3D scene flow recovery demonstrated.

Algorithms effective for rotation and non-rigid motion.

Theoretical analysis of performance limits using light field structure tensor.

Abstract

This paper presents novel techniques for recovering 3D dense scene flow, based on differential analysis of 4D light fields. The key enabling result is a per-ray linear equation, called the ray flow equation, that relates 3D scene flow to 4D light field gradients. The ray flow equation is invariant to 3D scene structure and applicable to a general class of scenes, but is under-constrained (3 unknowns per equation). Thus, additional constraints must be imposed to recover motion. We develop two families of scene flow algorithms by leveraging the structural similarity between ray flow and optical flow equations: local 'Lucas-Kanade' ray flow and global 'Horn-Schunck' ray flow, inspired by corresponding optical flow methods. We also develop a combined local-global method by utilizing the correspondence structure in the light fields. We demonstrate high precision 3D scene flow recovery for a wide range of scenarios, including rotation and non-rigid motion. We analyze the theoretical and practical performance limits of the proposed techniques via the light field structure tensor, a 3x3 matrix that encodes the local structure of light fields. We envision that the proposed analysis and algorithms will lead to design of future light-field cameras that are optimized for motion sensing, in addition to depth sensing.

Full text

Differential Scene Flow from Light Field Gradients:

Supplementary Technical Report

Sizhuo Ma   Brandon M. Smith   Mohit Gupta

Department of Computer Sciences,

University of Wisconsin-Madison, USA

{sizhuoma,bmsmith,mohitg}@cs.wisc.edu

In this report we show the full 3D visualization of the recovered motion for the experiments described in the main paper.