Leveraging Sparse LiDAR for RAFT-Stereo: A Depth Pre-Fill Perspective

TL;DR

This paper enhances stereo matching accuracy by pre-filling sparse LiDAR-guided disparity maps with interpolation, enabling effective LiDAR guidance even with very sparse point clouds, and introduces a combined approach called GRAFT-Stereo.

Contribution

It reveals why sparse LiDAR guidance degrades and proposes a simple pre-filling solution, leading to a new method that outperforms existing LiDAR-guided stereo techniques.

Findings

Pre-filling disparity maps with interpolation improves LiDAR guidance effectiveness.

Pre-filling is also beneficial when injecting LiDAR depth into image features via early fusion.

GRAFT-Stereo significantly outperforms existing methods under sparse LiDAR conditions.

Abstract

We investigate LiDAR guidance within the RAFT-Stereo framework, aiming to improve stereo matching accuracy by injecting precise LiDAR depth into the initial disparity map. We find that the effectiveness of LiDAR guidance drastically degrades when the LiDAR points become sparse (e.g., a few hundred points per frame), and we offer a novel explanation from a signal processing perspective. This insight leads to a surprisingly simple solution that enables LiDAR-guided RAFT-Stereo to thrive: pre-filling the sparse initial disparity map with interpolation. Interestingly, we find that pre-filling is also effective when injecting LiDAR depth into image features via early fusion, but for a fundamentally different reason, necessitating a distinct pre-filling approach. By combining both solutions, the proposed Guided RAFT-Stereo (GRAFT-Stereo) significantly outperforms existing LiDAR-guided methods under sparse LiDAR conditions across various datasets. We hope this study inspires more effective LiDAR-guided stereo methods.