AVS-Net: Audio-Visual Scale Net for Self-supervised Monocular Metric Depth Estimation

TL;DR

This paper introduces AVS-Net, a novel approach that leverages audio echoes to improve and scale-correct self-supervised monocular depth estimation, addressing generalization issues and eliminating the need for supervised depth data.

Contribution

The work demonstrates how integrating audio echoes can enhance depth prediction accuracy and enable scale correction in self-supervised monocular depth estimation.

Findings

Echoes help resolve object scale ambiguities.

The method improves state-of-the-art depth prediction models.

Echo-based supervision enables scale correction without supervised data.

Abstract

Metric depth prediction from monocular videos suffers from bad generalization between datasets and requires supervised depth data for scale-correct training. Self-supervised training using multi-view reconstruction can benefit from large scale natural videos but not provide correct scale, limiting its benefits. Recently, reflecting audible Echoes off objects is investigated for improved depth prediction and was shown to be sufficient to reconstruct objects at scale even without a visual signal. Because Echoes travel at fixed speed, they have the potential to resolve ambiguities in object scale and appearance. However, predicting depth end-to-end from sound and vision cannot benefit from unsupervised depth prediction approaches, which can process large scale data without sound annotation. In this work we show how Echoes can benefit depth prediction in two ways: When learning metric depth learned from supervised data and as supervisory signal for scale-correct self-supervised training. We show how we can improve the predictions of several state-of-the-art approaches and how the method can scale-correct a self-supervised depth approach.