Fully Exploiting Vision Foundation Model's Profound Prior Knowledge for Generalizable RGB-Depth Driving Scene Parsing

TL;DR

This paper introduces a novel Heterogeneous Feature Integration Transformer (HFIT) that leverages vision foundation models for improved RGB-depth driving scene parsing without re-training ViTs, demonstrating superior performance on benchmark datasets.

Contribution

The paper proposes a new HFIT architecture that exploits VFMs for RGB-depth scene parsing, utilizing inherent data characteristics and relative depth prediction to enhance generalization.

Findings

HFIT outperforms traditional and existing VFM-based scene parsing methods.

Relative depth prediction from VFMs effectively replaces depth maps.

The approach achieves state-of-the-art results on Cityscapes and KITTI datasets.

Abstract

Recent vision foundation models (VFMs), typically based on Vision Transformer (ViT), have significantly advanced numerous computer vision tasks. Despite their success in tasks focused solely on RGB images, the potential of VFMs in RGB-depth driving scene parsing remains largely under-explored. In this article, we take one step toward this emerging research area by investigating a feasible technique to fully exploit VFMs for generalizable RGB-depth driving scene parsing. Specifically, we explore the inherent characteristics of RGB and depth data, thereby presenting a Heterogeneous Feature Integration Transformer (HFIT). This network enables the efficient extraction and integration of comprehensive heterogeneous features without re-training ViTs. Relative depth prediction results from VFMs, used as inputs to the HFIT side adapter, overcome the limitations of the dependence on depth maps. Our proposed HFIT demonstrates superior performance compared to all other traditional single-modal and data-fusion scene parsing networks, pre-trained VFMs, and ViT adapters on the Cityscapes and KITTI Semantics datasets. We believe this novel strategy paves the way for future innovations in VFM-based data-fusion techniques for driving scene parsing. Our source code is publicly available at https://mias.group/HFIT.