Efficient Prediction of Dense Visual Embeddings via Distillation and RGB-D Transformers

TL;DR

This paper introduces DVEFormer, an efficient RGB-D Transformer model that predicts dense visual embeddings for robotics, enabling real-time semantic understanding and flexible text-based querying in indoor environments.

Contribution

The paper presents DVEFormer, a novel knowledge distillation approach that produces dense visual embeddings for semantic segmentation and natural language querying, optimized for real-time robotic applications.

Findings

Achieves 26.3 FPS on full model and 77.0 FPS on smaller variant.

Demonstrates competitive performance on indoor datasets.

Enables flexible text-based querying and 3D mapping integration.

Abstract

In domestic environments, robots require a comprehensive understanding of their surroundings to interact effectively and intuitively with untrained humans. In this paper, we propose DVEFormer - an efficient RGB-D Transformer-based approach that predicts dense text-aligned visual embeddings (DVE) via knowledge distillation. Instead of directly performing classical semantic segmentation with fixed predefined classes, our method uses teacher embeddings from Alpha-CLIP to guide our efficient student model DVEFormer in learning fine-grained pixel-wise embeddings. While this approach still enables classical semantic segmentation, e.g., via linear probing, it further enables flexible text-based querying and other applications, such as creating comprehensive 3D maps. Evaluations on common indoor datasets demonstrate that our approach achieves competitive performance while meeting real-time requirements, operating at 26.3 FPS for the full model and 77.0 FPS for a smaller variant on an NVIDIA Jetson AGX Orin. Additionally, we show qualitative results that highlight the effectiveness and possible use cases in real-world applications. Overall, our method serves as a drop-in replacement for traditional segmentation approaches while enabling flexible natural-language querying and seamless integration into 3D mapping pipelines for mobile robotics.