PoseSync: Robust pose based video synchronization

TL;DR

This paper introduces PoseSync, an end-to-end pose-based video synchronization method that uses pose detection and Dynamic Time Warping to align videos invariant to scale and shift, useful for performance evaluation and choreography.

Contribution

The paper presents a novel, scale and shift invariant pose matching pipeline for video synchronization using pose detection and DTW, applicable across multiple domains.

Findings

Effective pose matching across different videos

Robust synchronization despite scale and shift variations

Potential applications in sports and performance analysis

Abstract

Pose based video sychronization can have applications in multiple domains such as gameplay performance evaluation, choreography or guiding athletes. The subject's actions could be compared and evaluated against those performed by professionals side by side. In this paper, we propose an end to end pipeline for synchronizing videos based on pose. The first step crops the region where the person present in the image followed by pose detection on the cropped image. This is followed by application of Dynamic Time Warping(DTW) on angle/ distance measures between the pose keypoints leading to a scale and shift invariant pose matching pipeline.