CineMatte: Background Matting for Virtual Production and Beyond

TL;DR

CineMatte introduces a robust background matting framework for virtual production that employs a Siamese Vision Transformer with cross-attention, improving boundary detail recovery and generalization to real-world footage.

Contribution

The paper presents CineMatte, a novel background matting method using a cross-attention-conditioned ViT and introduces CineMatte-4K, a new high-resolution VP matting dataset.

Findings

CineMatte outperforms existing models on VP and real-world benchmarks.

The new dataset enables training and evaluation of VP matting in real-world conditions.

Replacing the detail branch with a pretrained feature upsampler reduces boundary artifacts.

Abstract

LED Virtual Production (VP) uses large LED volumes to render backgrounds in real time, enabling in-camera visual effects but making post-shot changes labor-intensive. We address this with CineMatte, a robust background matting framework for VP and beyond. CineMatte employs a cross-attention-conditioned design. Instead of concatenating the background with the input, CineMatte employs a Siamese, frozen DINOv3 Vision Transformer with shared weights to encode the input frame and the captured background separately. A cross-attention module compares the two streams to predict the foreground, preserving pretrained semantics and improving robustness to background shifts. Previous ViT-based matting models use a parallel convolutional "detail branch" to recover fine details, which can cause boundary artifacts in real-world samples due to semantic misalignment with the backbone. We instead replace it with a pretrained, image-guided feature upsampler, which largely mitigates the problem. We also introduce CineMatte-4K, a 4K HDR image-video dataset captured on a professional LED VP stage. To the best of our knowledge, the image subset is the first dataset for VP matting and is non-synthetic, obtained via green-screen insertion; the video subset includes camera motion with tracked trajectories so that arbitrary backgrounds can be rendered later with correct parallax. Across CineMatte-4K and public benchmarks (VideoMatte240K, YouTubeMatte), CineMatte not only excels in VP but also generalizes robustly to real-world footage.