PanMatch: Unleashing the Potential of Large Vision Models for Unified Matching Models

TL;DR

PanMatch introduces a unified large vision model capable of addressing various two-frame correspondence matching tasks within a single framework, eliminating the need for task-specific architectures and enabling zero-shot cross-view matching across diverse domains.

Contribution

The paper proposes PanMatch, a versatile foundation model that unifies multiple matching tasks using a single displacement estimation framework and a robust feature extractor, trained on a large cross-domain dataset.

Findings

Outperforms UniMatch and Flow-Anything on cross-task evaluations.

Achieves comparable results to state-of-the-art task-specific algorithms.

Demonstrates strong zero-shot performance in challenging scenarios like rainy weather and satellite images.

Abstract

This work presents PanMatch, a versatile foundation model for robust correspondence matching. Unlike previous methods that rely on task-specific architectures and domain-specific fine-tuning to support tasks like stereo matching, optical flow or feature matching, our key insight is that any two-frame correspondence matching task can be addressed within a 2D displacement estimation framework using the same model weights. Such a formulation eliminates the need for designing specialized unified architectures or task-specific ensemble models. Instead, it achieves multi-task integration by endowing displacement estimation algorithms with unprecedented generalization capabilities. To this end, we highlight the importance of a robust feature extractor applicable across multiple domains and tasks, and propose the feature transformation pipeline that leverage all-purpose features from Large Vision Models to endow matching baselines with zero-shot cross-view matching capabilities. Furthermore, we assemble a cross-domain dataset with near 1.8 million samples from stereo matching, optical flow, and feature matching domains to pretrain PanMatch. We demonstrate the versatility of PanMatch across a wide range of domains and downstream tasks using the same model weights. Our model outperforms UniMatch and Flow-Anything on cross-task evaluations, and achieves comparable performance to most state-of-the-art task-specific algorithms on task-oriented benchmarks. Additionally, PanMatch presents unprecedented zero-shot performance in abnormal scenarios, such as rainy day and satellite imagery, where most existing robust algorithms fail to yield meaningful results.