MT-Depth: Multi-task Instance feature analysis for the Depth Completion

TL;DR

This paper introduces an instance-aware depth completion framework that uses object masks from instance segmentation to improve depth prediction accuracy, especially around object boundaries and occlusions.

Contribution

The proposed method integrates instance masks with a depth completion network using cross-attention, enhancing depth accuracy without relying on dense semantic labels.

Findings

Achieves lower RMSE than baseline and previous methods on Virtual KITTI 2.

Improves depth accuracy near object boundaries and occlusions.

Maintains competitive MAE while enhancing overall depth quality.

Abstract

Depth completion plays a vital role in 3D perception systems, especially in scenarios where sparse depth data must be densified for tasks such as autonomous driving, robotics, and augmented reality. While many existing approaches rely on semantic segmentation to guide depth completion, they often overlook the benefits of object-level understanding. In this work, we introduce an instance-aware depth completion framework that explicitly integrates binary instance masks as spatial priors to refine depth predictions. Our model combines four main components: a frozen YOLO V11 instance segmentation branch, a U-Net-based depth completion backbone, a cross-attention fusion module, and an attention-guided prediction head. The instance segmentation branch generates per-image foreground masks that guide the depth branch via cross-attention, allowing the network to focus on object-centric regions during refinement. We validate our method on the Virtual KITTI 2 dataset, showing that it achieves lower Root Mean Squared Error (RMSE) compared to both a U-Net-only baseline and previous semantic-guided methods, while maintaining competitive Mean Absolute Error (MAE). Qualitative and quantitative results demonstrate that the proposed model effectively enhances depth accuracy near object boundaries, occlusions, and thin structures. Our findings suggest that incorporating instance-aware cues offers a promising direction for improving depth completion without relying on dense semantic labels.