AsyncMDE: Real-Time Monocular Depth Estimation via Asynchronous Spatial Memory

TL;DR

AsyncMDE introduces an asynchronous monocular depth estimation system that efficiently reuses features over time, enabling real-time, edge-deployable depth perception with high accuracy and low computational cost.

Contribution

It proposes a novel asynchronous framework combining a foundation model and a lightweight model for efficient, real-time depth estimation on edge devices.

Findings

Operates at 237 FPS on RTX 4090 with 3.83M parameters.

Recovers 77% of foundation model accuracy with 25X fewer parameters.

Achieves 161 FPS on Jetson AGX Orin, demonstrating real-time edge deployment.

Abstract

Foundation-model-based monocular depth estimation offers a viable alternative to active sensors for robot perception, yet its computational cost often prohibits deployment on edge platforms. Existing methods perform independent per-frame inference, wasting the substantial computational redundancy between adjacent viewpoints in continuous robot operation. This paper presents AsyncMDE, an asynchronous depth perception system consisting of a foundation model and a lightweight model that amortizes the foundation model's computational cost over time. The foundation model produces high-quality spatial features in the background, while the lightweight model runs asynchronously in the foreground, fusing cached memory with current observations through complementary fusion, outputting depth estimates, and autoregressively updating the memory. This enables cross-frame feature reuse with bounded accuracy degradation. At a mere 3.83M parameters, it operates at 237 FPS on an RTX 4090, recovering 77% of the accuracy gap to the foundation model while achieving a 25X parameter reduction. Validated across indoor static, dynamic, and synthetic extreme-motion benchmarks, AsyncMDE degrades gracefully between refreshes and achieves 161FPS on a Jetson AGX Orin with TensorRT, clearly demonstrating its feasibility for real-time edge deployment.