Synthetic Human Model Dataset for Skeleton Driven Non-rigid Motion Tracking and 3D Reconstruction

TL;DR

This paper presents a synthetic dataset designed for evaluating non-rigid 3D human reconstruction using RGB-D cameras, including detailed ground truth data for geometry and skeletons across multiple motion sequences.

Contribution

The authors introduce a comprehensive synthetic dataset with ground truth geometry, skeletons, and skinning weights for non-rigid human motion analysis.

Findings

Provides detailed ground truth data for 7 motion sequences

Enables evaluation of non-rigid 3D human reconstruction methods

Facilitates benchmarking with synthetic, controlled data

Abstract

We introduce a synthetic dataset for evaluating non-rigid 3D human reconstruction based on conventional RGB-D cameras. The dataset consist of seven motion sequences of a single human model. For each motion sequence per-frame ground truth geometry and ground truth skeleton are given. The dataset also contains skinning weights of the human model. More information about the dataset can be found at: https://research.csiro.au/robotics/our-work/databases/synthetic-human-model-dataset/

Equations

$$\left[\begin{array}{ccc}{f}_x& 0& c_x\\ 0& f_y& c_y\\ 0& 0& 1\end{array}\right]=\left[\begin{array}{ccc}1050& 0& 480\\ 0& 1050& 480\\ 0& 0& 1\end{array}\right].$$

Full text

Robotics and Autonomous Systems, CSIRO Data61, Brisbane, Australia

Queensland University of Technology, Brisbane, Australia

{shafeeq.elanattil, peyman.moghadam}@data61.csiro.au

1 Introduction

Volumetric 3D reconstruction for rigid scenes and objects is a well studied problem in computer vision and robotics [1, 2, 3]. Often reconstructed 3D maps are fused with other complementary modalities such as RGB information [4], non-visible imaging information such as thermal-infared [5, 6, 7] or sound [8] for application such as medical imaging [9], disaster response [10] and energy auditing [11].

The more general scenario, where the objects or scenes are dynamic and undergo non-rigid deformation, is still a challenge to be solved [12]. There are only a few publicly datasets available for evaluating RGB-D based non-rigid 3D reconstruction. Those datasets [13] are for general non-rigid subjects and not specific to human bodies. Even though the dataset published with [12] has the frame-to-frame live ground truth geometry and camera trajectory [14], they do not have ground truth skeleton joints and have very small non-rigid motion. We found that accurate skeleton joints play an important role in human performance capture algorithms. Motivated by this we developed a synthetic dataset which contains ground truth geometry and skeleton joints. The dataset contains human motion sequences which posses high frame-to-frame non-rigid motion.

2 Differences with previously published dataset

Our previous published dataset [14] has frame-to-frame live ground truth geometry and camera trajectory as the corresponding work [12] targets for usage of camera pose for non-rigid reconstruction. Frame-to-frame non-rigid motion in that sequences is very small. In the current publication [15] we are more focused on reconstructing non-rigid movements of human subjects like boxing, jumping etc.

These sequence posses high frame-to-frame non-rigid movements. This dataset also provides ground truth skeleton joints along with per frame ground truth geometry. In addition we are using a human model for motion tracking. The dataset also contains our human model with skinning weight information.

3 The Dataset

Our dataset [16] consists of

Ground truth of human 3D geometry at each frame in world coordinate frame. 2. 2.

Ground truth of skeleton points at each frame in world coordinate frame. 3. 3.

Extrinsic parameters of RGB and depth cameras 4. 4.

RGB and depth images.

The dataset consist of seven motion sequences of varying motions characteristics. Table 1 shows motion statistics of the corresponding data sequences. The motion is estimated as the sum of joint movement in each frame. We assign the same name as used in the CMU Mocap dataset for each sequence. The first two columns in Table 1 show name and number of frames in the sequence. The remaining columns shows the motion statistics for each data sequence.

Elanattil et al. [12] outlines the detail of the design and production of this synthetic dataset.

3.1 Data Description

The dataset consists of eight folders in which seven of them contains motion sequence data and remaining one contains our human model data. Each motion sequence folders named as the corresponding sequence name as shown in Table 1. Each sequence folder is structured as follows.

\dirtree.1 color. .1 depth. .1 gt. .1 skeleton. .1 transformation.txt.

Each motion sequence folder contains color, depth, gt, skeleton sub-folders and a transformation text file. The color and depth folders contains RGB and depth images respectively. The file names have the following form:

• •

frame_XXXXX.png: the RGB image of the scene;

• •

depth_XXXXX.png: the depth image;

The gt and skeleton folders contains ground truth mesh and ground truth skeleton joints correspond to each frame. The file name has the form:

• •

mesh_XXXXX.ply: mesh file of human in world co-ordinates;

• •

skeletonCSV_XXXXX.csv: skeleton joints of the human in camera co-ordinates;

where XXXXX is an integer number representing the frame number within the data sequence. In each skeleton file, joint locations of each skeleton bones are given. The transformation.txt file contains the world to camera transformation matrix.

3.2 Camera Parameters

In our synthetic environment the RGB and depth cameras are placed in same location and orientation. Therefor the extrinsic matrix between RGB and Depth cameras is $4\times 4$ identity matrix. The intrinsic parameter matrix for both RGB and depth images is

[TABLE]

3.3 Human Model

Our human model is a mesh with 17,021 vertices and 31,492 faces. Skinning bone indexes and weights of each vertices are given in labels.csv and weights.csv files of the model_information folder. Our model is based on 14 bone skeleton. The skeleton bone indexes and corresponding bone names are shown in Figure 1. The Figure 2 shows our human model which is color coded based on the highest skinning weight for the corresponding skeleton bone. Note that the color coding used for skeleton (Figure 1) and human model (Figure 2) is different.

3.4 Obtaining The Data

This dataset can be downloaded from https://research.csiro.au/robotics/our-work/databases/synthetic-human-model-dataset/. When making use of this data we ask that [15] [16] are cited.

3.5 Example Data Visualizations

Sample data from our dataset is shown in Figures 3 and 4 respectively. Figure 3 shows RGB image, depth image, ground truth mesh and 3D skeleton of a particular frame in ’Boxing’ motion sequence. Figure 4 shows RGB images of ’Boxing’, ’Exercise’ and ’Jumping’ motion sequences in the dataset.