3D model silhouette-based tracking in depth images for puppet suit dynamic video-mapping

TL;DR

This paper introduces a real-time 3D silhouette-based tracking method for puppets using depth images, enabling dynamic, accurate video-mapping of moving puppets on stage with non-flat surfaces.

Contribution

A novel real-time tracking approach using 3D model silhouettes in depth images, facilitating dynamic video-mapping of articulated objects like puppets.

Findings

Effective real-time tracking of puppet movements.

Successful dynamic video-mapping on non-flat surfaces.

Precise calibration enables coherent projection alignment.

Abstract

Video-mapping is the process of coherent video-projection of images, animations or movies on static objects or buildings for shows. This paper focuses on the dynamic video-mapping of the suit of a puppet being moved by its puppeteer on the theater stage. This may allow changing the costume dynamically and simulate light interaction and more. Contrary to common video-mapping, the image warping cannot be done once, offline, before the show. It must be done in real-time, and considering a non-flat projection surface, so that the video-projected suit always maps perfectly the puppet, automatically. Hence, we propose a new visual tracking method of articulated object, for the puppet tracking, exploiting the silhouette of a 3D model of it, in the depth images of a Kinect v2. Then, considering the precise calibration between the latter and the video-projector, that we propose, coherent dynamic video-mapping is made possible as the presented results show.