Enhancing Vision-Based Policies with Omni-View and Cross-Modality Knowledge Distillation for Mobile Robots

TL;DR

This paper introduces a knowledge distillation method that leverages omni-view depth information to improve vision-based navigation policies for lightweight mobile robots, enhancing transferability and efficiency.

Contribution

It presents a novel distillation approach transferring knowledge from an omni-view depth policy to a monocular policy, improving scene transfer and navigation performance.

Findings

Omni-view and depth inputs enhance scene transfer and navigation.

The distillation method improves monocular policy performance.

Real-world experiments validate the approach's effectiveness.

Abstract

Vision-based policies are widely applied in robotics for tasks such as manipulation and locomotion. On lightweight mobile robots, however, they face a trilemma of limited scene transferability, restricted onboard computation resources, and sensor hardware cost. To address these issues, we propose a knowledge distillation approach that transfers knowledge from an information-rich, appearance invariant omniview depth policy to a lightweight monocular policy. The key idea is to train the student not only to mimic the expert actions but also to align with the latent embeddings of the omni view depth teacher. Experiments demonstrate that omni-view and depth inputs improve the scene transfer and navigation performance, and that the proposed distillation method enhances the performance of a singleview monocular policy, compared with policies solely imitating actions. Real world experiments further validate the effectiveness and practicality of our approach. Code will be released publicly.