Leveraging Large (Visual) Language Models for Robot 3D Scene Understanding

TL;DR

This paper explores using large pre-trained language models, alone or combined with vision, to improve semantic 3D scene understanding in robotics, achieving around 70% accuracy and better generalization.

Contribution

It introduces and compares various language-based and vision-language paradigms for scene classification, demonstrating their effectiveness over traditional vision-only methods.

Findings

Language-based methods achieve ~70% accuracy in room classification.

These methods outperform pure vision and graph classifiers.

Language models show strong generalization and transfer capabilities.

Abstract

Abstract semantic 3D scene understanding is a problem of critical importance in robotics. As robots still lack the common-sense knowledge about household objects and locations of an average human, we investigate the use of pre-trained language models to impart common sense for scene understanding. We introduce and compare a wide range of scene classification paradigms that leverage language only (zero-shot, embedding-based, and structured-language) or vision and language (zero-shot and fine-tuned). We find that the best approaches in both categories yield $\sim 70\%$ room classification accuracy, exceeding the performance of pure-vision and graph classifiers. We also find such methods demonstrate notable generalization and transfer capabilities stemming from their use of language.