Learning Semantic Traversability with Egocentric Video and Automated Annotation Strategy

TL;DR

This paper introduces a scalable method for training semantic traversability estimators for autonomous robots using egocentric videos and automated annotation, reducing manual effort and improving generalization across urban environments.

Contribution

The authors propose a novel automated annotation strategy leveraging foundation models and egocentric videos, enabling scalable training of semantic traversability estimators without manual labeling.

Findings

High accuracy in diverse urban scenarios

Effective handling of various camera viewpoints

Demonstrated real-world deployment success

Abstract

For reliable autonomous robot navigation in urban settings, the robot must have the ability to identify semantically traversable terrains in the image based on the semantic understanding of the scene. This reasoning ability is based on semantic traversability, which is frequently achieved using semantic segmentation models fine-tuned on the testing domain. This fine-tuning process often involves manual data collection with the target robot and annotation by human labelers which is prohibitively expensive and unscalable. In this work, we present an effective methodology for training a semantic traversability estimator using egocentric videos and an automated annotation process. Egocentric videos are collected from a camera mounted on a pedestrian's chest. The dataset for training the semantic traversability estimator is then automatically generated by extracting semantically traversable regions in each video frame using a recent foundation model in image segmentation and its prompting technique. Extensive experiments with videos taken across several countries and cities, covering diverse urban scenarios, demonstrate the high scalability and generalizability of the proposed annotation method. Furthermore, performance analysis and real-world deployment for autonomous robot navigation showcase that the trained semantic traversability estimator is highly accurate, able to handle diverse camera viewpoints, computationally light, and real-world applicable. The summary video is available at https://youtu.be/EUVoH-wA-lA.