SHED Light on Segmentation for Dense Prediction

TL;DR

SHED introduces a hierarchy-aware encoder-decoder architecture that explicitly incorporates segmentation to improve dense prediction tasks like depth estimation and 3D reconstruction, enhancing boundary sharpness, coherence, and interpretability.

Contribution

The paper presents SHED, a novel architecture that integrates hierarchical segmentation reasoning into dense prediction, improving structural consistency and cross-domain generalization without explicit segmentation supervision.

Findings

Improves depth boundary sharpness and segment coherence.

Enhances 3D reconstruction quality and interpretability.

Demonstrates strong cross-domain generalization from synthetic to real-world data.

Abstract

Dense prediction infers per-pixel values from a single image and is fundamental to 3D perception and robotics. Although real-world scenes exhibit strong structure, existing methods treat it as an independent pixel-wise prediction, often resulting in structural inconsistencies. We propose SHED, a novel encoder-decoder architecture that enforces geometric prior explicitly by incorporating segmentation into dense prediction. By bidirectional hierarchical reasoning, segment tokens are hierarchically pooled in the encoder and unpooled in the decoder to reverse the hierarchy. The model is supervised only at the final output, allowing the segment hierarchy to emerge without explicit segmentation supervision. SHED improves depth boundary sharpness and segment coherence, while demonstrating strong cross-domain generalization from synthetic to the real-world environments. Its hierarchy-aware decoder better captures global 3D scene layouts, leading to improved semantic segmentation performance. Moreover, SHED enhances 3D reconstruction quality and reveals interpretable part-level structures that are often missed by conventional pixel-wise methods.