Capture Stage Matting: Challenges, Approaches, and Solutions for Offline and Real-Time Processing

TL;DR

This paper discusses the unique challenges of matting in capture stage recordings, reviews existing approaches, and proposes an efficient pipeline for improved offline and real-time processing with validation using a diffusion model.

Contribution

It provides a comprehensive analysis of capture stage matting challenges and introduces an adaptable, annotation-efficient pipeline for enhanced offline and real-time matting.

Findings

Identified key peculiarities of capture stage content affecting matting quality.

Proposed a workflow that improves matting performance without extensive annotations.

Validated the approach using a diffusion model for objective assessment.

Abstract

Capture stages are high-end sources of state-of-the-art recordings for downstream applications in movies, games, and other media. One crucial step in almost all pipelines is matting, i.e., separating captured performances from the background. While common matting algorithms deliver remarkable performance in other applications like teleconferencing and mobile entertainment, we found that they struggle significantly with the peculiarities of capture stage content. The goal of our work is to share insights into those challenges as a curated list of these characteristics along with a constructive discussion for proactive intervention and present a guideline to practitioners for an improved workflow to mitigate unresolved challenges. To this end, we also demonstrate an efficient pipeline to adapt state-of-the-art approaches to such custom setups without the need for extensive annotations, both offline and real-time. For an objective evaluation, we introduce a validation methodology using a state-of-the-art diffusion model to demonstrate the benefits of our approach.