Occlusion Boundary and Depth: Mutual Enhancement via Multi-Task Learning

TL;DR

This paper introduces MoDOT, a multi-task learning framework that jointly estimates occlusion boundaries and depth from a single image, leveraging their mutual relationship to improve accuracy and generalization.

Contribution

The paper proposes a novel multi-task framework with a Cross-Attention Strip Module and a geometric consistency loss for joint occlusion boundary and depth estimation, along with a new dataset OB-Hypersim.

Findings

MoDOT outperforms single-task and multi-task baselines on synthetic datasets and NYUD-v2.

Models trained on synthetic data generalize well to real-world scenes without fine-tuning.

Joint modeling produces sharper boundaries and better geometric fidelity in depth maps.

Abstract

Occlusion Boundary Estimation (OBE) identifies boundaries arising from both inter-object occlusions and self-occlusion within individual objects. This task is closely related to Monocular Depth Estimation (MDE), which infers depth from a single image, as Occlusion Boundaries (OBs) provide critical geometric cues for resolving depth ambiguities, while depth can conversely refine occlusion reasoning. In this paper, we aim to systematically model and exploit this mutually beneficial relationship. To this end, we propose MoDOT, a novel framework for joint estimation of depth and OBs, which incorporates a new Cross-Attention Strip Module (CASM) to leverage mid-level OB features for depth prediction, and a novel OB-Depth Constraint Loss (OBDCL) to enforce geometric consistency. To facilitate this study, we contribute OB-Hypersim, a large-scale photorealistic dataset with precise depth and self-occlusion-handled OB annotations. Extensive experiments on two synthetic datasets and NYUD-v2 demonstrate that MoDOT achieves significantly better performance than single-task baselines and multi-task competitors. Furthermore, models trained solely on our synthetic data demonstrate strong generalization to real-world scenes without fine-tuning, producing depth maps with sharper boundaries and improved geometric fidelity. Collectively, these results underscore the significant benefits of jointly modeling OBs and depth. Code and resources are available at https://github.com/xul-ops/MoDOT.