SIDQL: An Efficient Keyframe Extraction and Motion Reconstruction Framework in Motion Capture

TL;DR

This paper introduces SIDQL, a deep reinforcement learning framework for efficient keyframe extraction and motion reconstruction in motion capture, significantly reducing data volume and latency while maintaining high reconstruction accuracy.

Contribution

The paper presents a novel SIDQL framework that uses Deep Q-Learning for optimal keyframe selection and motion reconstruction in spherical coordinates, improving efficiency over existing methods.

Findings

Reduces data volume and transmission latency.

Maintains low reconstruction error (<0.09) with five keyframes.

Effective on CMU motion database.

Abstract

Metaverse, which integrates the virtual and physical worlds, has emerged as an innovative paradigm for changing people's lifestyles. Motion capture has become a reliable approach to achieve seamless synchronization of the movements between avatars and human beings, which plays an important role in diverse Metaverse applications. However, due to the continuous growth of data, current communication systems face a significant challenge of meeting the demand of ultra-low latency during application. In addition, current methods also have shortcomings when selecting keyframes, e.g., relying on recognizing motion types and artificially selected keyframes. Therefore, the utilization of keyframe extraction and motion reconstruction techniques could be considered a feasible and promising solution. In this work, a new motion reconstruction algorithm is designed in a spherical coordinate system involving location and velocity information. Then, we formalize the keyframe extraction problem into an optimization problem to reduce the reconstruction error. Using Deep Q-Learning (DQL), the Spherical Interpolation based Deep Q-Learning (SIDQL) framework is proposed to generate proper keyframes for reconstructing the motion sequences. We use the CMU database to train and evaluate the framework. Our scheme can significantly reduce the data volume and transmission latency compared to various baselines while maintaining a reconstruction error of less than 0.09 when extracting five keyframes.