A Flexible Framework for Virtual Omnidirectional Vision to Improve Operator Situation Awareness

TL;DR

This paper introduces a flexible, multi-camera virtual projection framework that enhances operator situation awareness in teleoperated robots by fusing camera and Lidar data, demonstrated on a compact omnidirectional system and the Boston Dynamics Spot.

Contribution

It presents a novel multi-camera fusion method for virtual projections and scene understanding, reducing system complexity and broadening application scope.

Findings

Improved operator awareness with virtual projections.

Effective fusion of camera images and Lidar data.

Successful implementation on compact omnidirectional camera and Boston Dynamics Spot.

Abstract

During teleoperation of a mobile robot, providing good operator situation awareness is a major concern as a single mistake can lead to mission failure. Camera streams are widely used for teleoperation but offer limited field-of-view. In this paper, we present a flexible framework for virtual projections to increase situation awareness based on a novel method to fuse multiple cameras mounted anywhere on the robot. Moreover, we propose a complementary approach to improve scene understanding by fusing camera images and geometric 3D Lidar data to obtain a colorized point cloud. The implementation on a compact omnidirectional camera reduces system complexity considerably and solves multiple use-cases on a much smaller footprint compared to traditional approaches such as actuated pan-tilt units. Finally, we demonstrate the generality of the approach by application to the multi-camera system of the Boston Dynamics Spot. The software implementation is available as open-source ROS packages on the project page https://tu-darmstadt-ros-pkg.github.io/omnidirectional_vision.