SmartPortraits: Depth Powered Handheld Smartphone Dataset of Human Portraits for State Estimation, Reconstruction and Synthesis

TL;DR

This paper introduces a comprehensive dataset of 1000 smartphone-recorded human portrait videos with synchronized depth data, aimed at advancing state estimation, 3D reconstruction, and view synthesis in uncontrolled conditions.

Contribution

It provides a novel, large-scale dataset combining smartphone video, IMU, and depth data, along with benchmark evaluations for camera alignment and potential applications.

Findings

Benchmark results for camera alignment methods

Validation of dataset quality with motion capture comparison

Potential for improved 3D reconstruction and view synthesis

Abstract

We present a dataset of 1000 video sequences of human portraits recorded in real and uncontrolled conditions by using a handheld smartphone accompanied by an external high-quality depth camera. The collected dataset contains 200 people captured in different poses and locations and its main purpose is to bridge the gap between raw measurements obtained from a smartphone and downstream applications, such as state estimation, 3D reconstruction, view synthesis, etc. The sensors employed in data collection are the smartphone's camera and Inertial Measurement Unit (IMU), and an external Azure Kinect DK depth camera software synchronized with sub-millisecond precision to the smartphone system. During the recording, the smartphone flash is used to provide a periodic secondary source of lightning. Accurate mask of the foremost person is provided as well as its impact on the camera alignment accuracy. For evaluation purposes, we compare multiple state-of-the-art camera alignment methods by using a Motion Capture system. We provide a smartphone visual-inertial benchmark for portrait capturing, where we report results for multiple methods and motivate further use of the provided trajectories, available in the dataset, in view synthesis and 3D reconstruction tasks.