BATMAN: Bilateral Attention Transformer in Motion-Appearance Neighboring Space for Video Object Segmentation

TL;DR

BATMAN introduces a novel bilateral attention transformer that leverages motion and appearance cues, along with optical flow calibration, to significantly improve semi-supervised video object segmentation performance across multiple benchmarks.

Contribution

The paper proposes a new Bilateral Attention Transformer with optical flow calibration for better segmentation of similar objects in close proximity.

Findings

Outperforms all existing state-of-the-art methods on four VOS benchmarks.

Effectively captures motion and appearance for improved segmentation.

Reduces noise at object boundaries through optical flow calibration.

Abstract

Video Object Segmentation (VOS) is fundamental to video understanding. Transformer-based methods show significant performance improvement on semi-supervised VOS. However, existing work faces challenges segmenting visually similar objects in close proximity of each other. In this paper, we propose a novel Bilateral Attention Transformer in Motion-Appearance Neighboring space (BATMAN) for semi-supervised VOS. It captures object motion in the video via a novel optical flow calibration module that fuses the segmentation mask with optical flow estimation to improve within-object optical flow smoothness and reduce noise at object boundaries. This calibrated optical flow is then employed in our novel bilateral attention, which computes the correspondence between the query and reference frames in the neighboring bilateral space considering both motion and appearance. Extensive experiments validate the effectiveness of BATMAN architecture by outperforming all existing state-of-the-art on all four popular VOS benchmarks: Youtube-VOS 2019 (85.0%), Youtube-VOS 2018 (85.3%), DAVIS 2017Val/Testdev (86.2%/82.2%), and DAVIS 2016 (92.5%).