Beyond Appearance: Geometric Cues for Robust Video Instance Segmentation

TL;DR

This paper introduces geometric cues, specifically monocular depth estimation, to improve the robustness of Video Instance Segmentation against occlusions and motion blur, achieving state-of-the-art results.

Contribution

It systematically explores three methods to incorporate depth information into VIS, demonstrating significant improvements with two of these approaches.

Findings

EDC and SV methods significantly improve VIS robustness

EDC achieves 56.2 AP with Swin-L backbone on OVIS benchmark

Depth cues are validated as critical for robust video understanding

Abstract

Video Instance Segmentation (VIS) fundamentally struggles with pervasive challenges including object occlusions, motion blur, and appearance variations during temporal association. To overcome these limitations, this work introduces geometric awareness to enhance VIS robustness by strategically leveraging monocular depth estimation. We systematically investigate three distinct integration paradigms. Expanding Depth Channel (EDC) method concatenates the depth map as input channel to segmentation networks; Sharing ViT (SV) designs a uniform ViT backbone, shared between depth estimation and segmentation branches; Depth Supervision (DS) makes use of depth prediction as an auxiliary training guide for feature learning. Though DS exhibits limited effectiveness, benchmark evaluations demonstrate that EDC and SV significantly enhance the robustness of VIS. When with Swin-L backbone, our EDC method gets 56.2 AP, which sets a new state-of-the-art result on OVIS benchmark. This work conclusively establishes depth cues as critical enablers for robust video understanding.