DeFM: Learning Foundation Representations from Depth for Robotics

TL;DR

DeFM is a self-supervised foundation model trained on 60 million depth images that learns geometric and semantic representations, enabling robust robotic perception and manipulation across diverse environments without task-specific fine-tuning.

Contribution

This work introduces DeFM, the first large-scale self-supervised foundation model for depth images, with novel normalization and distillation techniques for robotic applications.

Findings

Achieves state-of-the-art results on multiple depth-based benchmarks

Demonstrates strong generalization from simulation to real-world environments

Provides pretrained models for off-the-shelf robotic depth perception

Abstract

Depth sensors are widely deployed across robotic platforms, and advances in fast, high-fidelity depth simulation have enabled robotic policies trained on depth observations to achieve robust sim-to-real transfer for a wide range of tasks. Despite this, representation learning for depth modality remains underexplored compared to RGB, where large-scale foundation models now define the state of the art. To address this gap, we present DeFM, a self-supervised foundation model trained entirely on depth images for robotic applications. Using a DINO-style self-distillation objective on a curated dataset of 60M depth images, DeFM learns geometric and semantic representations that generalize to diverse environments, tasks, and sensors. To retain metric awareness across multiple scales, we introduce a novel input normalization strategy. We further distill DeFM into compact models suitable for resource-constrained robotic systems. When evaluated on depth-based classification, segmentation, navigation, locomotion, and manipulation benchmarks, DeFM achieves state-of-the-art performance and demonstrates strong generalization from simulation to real-world environments. We release all our pretrained models, which can be adopted off-the-shelf for depth-based robotic learning without task-specific fine-tuning. Webpage: https://de-fm.github.io/