GraftNet: Towards Domain Generalized Stereo Matching with a Broad-Spectrum and Task-Oriented Feature

TL;DR

This paper introduces GraftNet, a stereo matching model that enhances domain generalization by grafting broad-spectrum, task-oriented features from large-scale datasets into existing architectures, significantly improving transferability across diverse datasets.

Contribution

It proposes a novel feature grafting method using cosine similarity to incorporate broad-spectrum features into stereo matching networks, enhancing their domain generalization capabilities.

Findings

Outperforms existing methods on SceneFlow to KITTI and Middlebury datasets.

Significantly improves generalization from synthetic to real-world data.

Compatible with architectures like PSMNet and GANet.

Abstract

Although supervised deep stereo matching networks have made impressive achievements, the poor generalization ability caused by the domain gap prevents them from being applied to real-life scenarios. In this paper, we propose to leverage the feature of a model trained on large-scale datasets to deal with the domain shift since it has seen various styles of images. With the cosine similarity based cost volume as a bridge, the feature will be grafted to an ordinary cost aggregation module. Despite the broad-spectrum representation, such a low-level feature contains much general information which is not aimed at stereo matching. To recover more task-specific information, the grafted feature is further input into a shallow network to be transformed before calculating the cost. Extensive experiments show that the model generalization ability can be improved significantly with this broad-spectrum and task-oriented feature. Specifically, based on two well-known architectures PSMNet and GANet, our methods are superior to other robust algorithms when transferring from SceneFlow to KITTI 2015, KITTI 2012, and Middlebury. Code is available at https://github.com/SpadeLiu/Graft-PSMNet.