VirtualFencer: Generating Fencing Bouts based on Strategies Extracted from In-the-Wild Videos

TL;DR

VirtualFencer is a system that extracts 3D fencing motions and strategies from in-the-wild videos without supervision, enabling realistic fencing behavior generation and interactive fencing scenarios.

Contribution

It introduces a novel data-driven approach to model fencing strategies and motions directly from unannotated videos, facilitating realistic and interactive fencing simulations.

Findings

Successfully extracts 3D fencing motions from videos

Generates realistic fencing behavior for various scenarios

Enables interactive fencing against professionals

Abstract

Fencing is a sport where athletes engage in diverse yet strategically logical motions. While most motions fall into a few high-level actions (e.g. step, lunge, parry), the execution can vary widely-fast vs. slow, large vs. small, offensive vs. defensive. Moreover, a fencer's actions are informed by a strategy that often comes in response to the opponent's behavior. This combination of motion diversity with underlying two-player strategy motivates the application of data-driven modeling to fencing. We present VirtualFencer, a system capable of extracting 3D fencing motion and strategy from in-the-wild video without supervision, and then using that extracted knowledge to generate realistic fencing behavior. We demonstrate the versatile capabilities of our system by having it (i) fence against itself (self-play), (ii) fence against a real fencer's motion from online video, and (iii) fence interactively against a professional fencer.