Temporal Upsampling of Depth Maps Using a Hybrid Camera

TL;DR

This paper introduces a hybrid camera system combining high-frame-rate video and low-frame-rate depth cameras, along with a novel algorithm for interpolating intermediate depth maps and scene flow, enabling better capture of fast, non-rigid motions.

Contribution

It presents a new hybrid camera setup and a novel algorithm for temporal upsampling of depth maps using auxiliary color images, improving motion capture accuracy.

Findings

Scene flow estimation is more precise than tracking-based methods.

The approach outperforms state-of-the-art techniques.

Effective for fast, non-rigid motions.

Abstract

In recent years, consumer-level depth cameras have been adopted for various applications. However, they often produce depth maps at only a moderately high frame rate (approximately 30 frames per second), preventing them from being used for applications such as digitizing human performance involving fast motion. On the other hand, low-cost, high-frame-rate video cameras are available. This motivates us to develop a hybrid camera that consists of a high-frame-rate video camera and a low-frame-rate depth camera and to allow temporal interpolation of depth maps with the help of auxiliary color images. To achieve this, we develop a novel algorithm that reconstructs intermediate depth maps and estimates scene flow simultaneously. We test our algorithm on various examples involving fast, non-rigid motions of single or multiple objects. Our experiments show that our scene flow estimation method is more precise than a tracking-based method and the state-of-the-art techniques.