U(PM)$^2$:Unsupervised polygon matching with pre-trained models for challenging stereo images

TL;DR

This paper introduces U(PM)$^2$, an unsupervised, low-cost polygon matching method using pre-trained models that effectively handles challenging stereo images with disparity discontinuities and scale variations.

Contribution

It presents a novel pipeline combining pre-trained segmentation and feature extraction with a multi-stage matching strategy for polygon correspondence.

Findings

Achieved state-of-the-art accuracy on ScanNet and SceneFlow datasets.

Demonstrated strong generalization with low computational cost.

Operates without any training requirement.

Abstract

Stereo image matching is a fundamental task in computer vision, photogrammetry and remote sensing, but there is an almost unexplored field, i.e., polygon matching, which faces the following challenges: disparity discontinuity, scale variation, training requirement, and generalization. To address the above-mentioned issues, this paper proposes a novel U(PM)$^2$: low-cost unsupervised polygon matching with pre-trained models by uniting automatically learned and handcrafted features, of which pipeline is as follows: firstly, the detector leverages the pre-trained segment anything model to obtain masks; then, the vectorizer converts the masks to polygons and graphic structure; secondly, the global matcher addresses challenges from global viewpoint changes and scale variation based on bidirectional-pyramid strategy with pre-trained LoFTR; finally, the local matcher further overcomes local disparity discontinuity and topology inconsistency of polygon matching by local-joint geometry and multi-feature matching strategy with Hungarian algorithm. We benchmark our U(PM)$^2$ on the ScanNet and SceneFlow datasets using our proposed new metric, which achieved state-of-the-art accuracy at a competitive speed and satisfactory generalization performance at low cost without any training requirement.