DOMR: Establishing Cross-View Segmentation via Dense Object Matching

TL;DR

DOMR introduces a dense object matching framework that effectively establishes cross-view object correspondences between egocentric and exocentric views by leveraging positional, semantic, and inter-object relationships.

Contribution

The paper presents the DOMR framework with a novel dense object matcher and mask refinement, achieving state-of-the-art results in cross-view segmentation tasks.

Findings

Achieves a mean IoU of 49.7% on Ego→Exo

Achieves a mean IoU of 55.2% on Exo→Ego

Outperforms previous methods by 4.3-5.8% in IoU

Abstract

Cross-view object correspondence involves matching objects between egocentric (first-person) and exocentric (third-person) views. It is a critical yet challenging task for visual understanding. In this work, we propose the Dense Object Matching and Refinement (DOMR) framework to establish dense object correspondences across views. The framework centers around the Dense Object Matcher (DOM) module, which jointly models multiple objects. Unlike methods that directly match individual object masks to image features, DOM leverages both positional and semantic relationships among objects to find correspondences. DOM integrates a proposal generation module with a dense matching module that jointly encodes visual, spatial, and semantic cues, explicitly constructing inter-object relationships to achieve dense matching among objects. Furthermore, we combine DOM with a mask refinement head designed to improve the completeness and accuracy of the predicted masks, forming the complete DOMR framework. Extensive evaluations on the Ego-Exo4D benchmark demonstrate that our approach achieves state-of-the-art performance with a mean IoU of 49.7% on Ego$\to$Exo and 55.2% on Exo$\to$Ego. These results outperform those of previous methods by 5.8% and 4.3%, respectively, validating the effectiveness of our integrated approach for cross-view understanding.