Motion-Based Sign Language Video Summarization using Curvature and Torsion

TL;DR

This paper introduces a novel 3-D motion analysis method using curvature and torsion to improve sign language video summarization, leading to more accurate keyframe extraction and better understanding of sign language videos.

Contribution

It extends existing 2-D curvature-based methods to 3-D, proposing a new informative function for keyframe detection in sign language videos.

Findings

Effective keyframe identification using 3-D curvature and torsion.

Improved sign language video summarization performance.

Promising results in gloss classification and understanding tasks.

Abstract

An interesting problem in many video-based applications is the generation of short synopses by selecting the most informative frames, a procedure which is known as video summarization. For sign language videos the benefits of using the $t$-parameterized counterpart of the curvature of the 2-D signer's wrist trajectory to identify keyframes, have been recently reported in the literature. In this paper we extend these ideas by modeling the 3-D hand motion that is extracted from each frame of the video. To this end we propose a new informative function based on the $t$-parameterized curvature and torsion of the 3-D trajectory. The method to characterize video frames as keyframes depends on whether the motion occurs in 2-D or 3-D space. Specifically, in the case of 3-D motion we look for the maxima of the harmonic mean of the curvature and torsion of the target's trajectory; in the planar motion case we seek for the maxima of the trajectory's curvature. The proposed 3-D feature is experimentally evaluated in applications of sign language videos on (1) objective measures using ground-truth keyframe annotations, (2) human-based evaluation of understanding, and (3) gloss classification and the results obtained are promising.