High Fidelity Interactive Video Segmentation Using Tensor Decomposition Boundary Loss Convolutional Tessellations and Context Aware Skip Connections

TL;DR

HyperSeg is a high fidelity interactive video segmentation algorithm that maintains high resolution features and improves accuracy using tensor decomposition, tessellation, and boundary loss, suitable for VFX and medical imaging.

Contribution

The paper introduces HyperSeg, a novel high-resolution video segmentation method employing tensor decomposition, tessellation, and boundary loss for improved fidelity and temporal coherence.

Findings

Demonstrates superior accuracy over baseline models on high-resolution video data.

Introduces the VFX Segmentation Dataset with over 27,000 annotated frames.

Achieves high fidelity segmentation without downsampling or pooling.

Abstract

We provide a high fidelity deep learning algorithm (HyperSeg) for interactive video segmentation tasks using a convolutional network with context-aware skip connections, and compressed, hypercolumn image features combined with a convolutional tessellation procedure. In order to maintain high output fidelity, our model crucially processes and renders all image features in high resolution, without utilizing downsampling or pooling procedures. We maintain this consistent, high grade fidelity efficiently in our model chiefly through two means: (1) We use a statistically-principled tensor decomposition procedure to modulate the number of hypercolumn features and (2) We render these features in their native resolution using a convolutional tessellation technique. For improved pixel level segmentation results, we introduce a boundary loss function; for improved temporal coherence in video data, we include temporal image information in our model. Through experiments, we demonstrate the improved accuracy of our model against baseline models for interactive segmentation tasks using high resolution video data. We also introduce a benchmark video segmentation dataset, the VFX Segmentation Dataset, which contains over 27,046 high resolution video frames, including greenscreen and various composited scenes with corresponding, hand crafted, pixel level segmentations. Our work presents an extension to improvement to state of the art segmentation fidelity with high resolution data and can be used across a broad range of application domains, including VFX pipelines and medical imaging disciplines.