EAGLE: Efficient Adaptive Geometry-based Learning in Cross-view Understanding

TL;DR

This paper proposes a novel unsupervised cross-view adaptation method for semantic scene understanding that models geometric structural changes across camera views, achieving state-of-the-art results.

Contribution

It introduces a cross-view geometric constraint, a geodesic flow-based correlation metric, and a view-condition prompting mechanism for improved cross-view semantic segmentation.

Findings

Achieves state-of-the-art performance on cross-view benchmarks.

Effectively models geometric structural changes across camera views.

Outperforms prior unsupervised domain adaptation and open-vocabulary methods.

Abstract

Unsupervised Domain Adaptation has been an efficient approach to transferring the semantic segmentation model across data distributions. Meanwhile, the recent Open-vocabulary Semantic Scene understanding based on large-scale vision language models is effective in open-set settings because it can learn diverse concepts and categories. However, these prior methods fail to generalize across different camera views due to the lack of cross-view geometric modeling. At present, there are limited studies analyzing cross-view learning. To address this problem, we introduce a novel Unsupervised Cross-view Adaptation Learning approach to modeling the geometric structural change across views in Semantic Scene Understanding. First, we introduce a novel Cross-view Geometric Constraint on Unpaired Data to model structural changes in images and segmentation masks across cameras. Second, we present a new Geodesic Flow-based Correlation Metric to efficiently measure the geometric structural changes across camera views. Third, we introduce a novel view-condition prompting mechanism to enhance the view-information modeling of the open-vocabulary segmentation network in cross-view adaptation learning. The experiments on different cross-view adaptation benchmarks have shown the effectiveness of our approach in cross-view modeling, demonstrating that we achieve State-of-the-Art (SOTA) performance compared to prior unsupervised domain adaptation and open-vocabulary semantic segmentation methods.